We are in love with CoreOS and its environment (Docker, Systemd, Fleet) and even though we have seen many blog posts all over the internet about it, we haven't seen anyone using CoreOS on a Jenkins Slave so far. Why not? On this blog post, we are sharing our setup guide, I hope you find it useful!

Old jenkins setup

Before I start talking about what we achieved, let's talk a little bit about what we had:

1. We had a jenkins server running on a m1.medium server with ubuntu installed. This server was used not only to run jenkins but to also run the tests for all our apps.

2. We could not run tests in parallel and this, specially when merging branches, caused a long queue of builds.

Our Goal

Reliable jenkins builds

One of the most common issues with any CI server is having a test that fails on CI but not on your machine. As one image generated using docker can be used in different machines (e.g. your CI server and your machine), docker alone improves your test reliability.

Parallel builds on jenkins

Parallel builds is very important as when multiple branches are merged or commits are pushed, having a fast reply from our CI server is necessary to ship code as fast as possible.

New jenkins setup

Amazon EC2 Plugin

In order to boot a slave on Amazon EC2 every time a new build needs to be executed, we will be using the Amazon EC2 Plugin.

Permission Issues

Unfortunately, we had permission issues after following the plugins instructions and we had to use the following IAM permissions instead:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:DescribeRegions"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Action": [
                "ec2:CreateTags",
                "ec2:DescribeImages",
                "ec2:DescribeImageAttribute",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceAttribute",
                "ec2:DescribeKeyPairs",
                "ec2:GetConsoleOutput",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "ec2:Region": "us-west-2"
                }
            }
        }
    ]
}

Note that we are limiting this user actions to us-west-2 region only. We are doing this because the EC2 Key Pair's Private Key has to be given to the plugin and while our Jenkins is not in our VPC we don't want to risk our other systems.

One other issue that we haven't solved yet is that the plugin's Check AMI button is not working. Thanks to @sleepingkyoto, we have updated the IAM permissions above and fixed this issue.

CoreOS Setup

CoreOS is an OS that's designed to be a modern, minimal base to build your platform. Consumes 50% less RAM on boot than an average Linux installation.

We are using the AMIs provided by CoreOS to setup our slaves as they come installed with Docker. To configure our slaves, we run the following init script.

#!/bin/sh

# Install Java
mkdir /home/core/java
cd /home/core/java
wget -O java.tar.gz http://javadl.sun.com/webapps/download/AutoDL?BundleId=83376
tar xzvf java.tar.gz
rm java.tar.gz
rm /home/core/.bashrc
echo 'export PATH=$PATH:/home/core/java/jre1.7.0_51/bin/' > /home/core/.bashrc

# HACK: Copy github ssh key
echo "-----BEGIN RSA PRIVATE KEY-----
... REST OF THE KEY ...
-----END RSA PRIVATE KEY-----" > /home/core/.ssh/id_rsa
chmod 600 /home/core/.ssh/id_rsa

# Add github to known_hosts
ssh-keyscan -H github.com > /home/core/.ssh/known_hosts

There are some hacks on the script above. When installing Java, for instance, we are removing the content of .bashrc provided by CoreOS and this could cause a bug in a future release.

Conclusion

We configure (or start a paused slave) whenever a new build is to be executed. As each slave manages its own dependencies (ruby/postgres/redis/etc.) we now run the builds in parallel.

This also means we can spin powerful slaves while having a m1.small jenkins server running all of the time.

I didn't explain how to build your own Dockerfile and how our CI build script looks like, as these topics have been discussed in other blogs before, but I intend to give more details about it in future posts.

Once we start using docker for our staging and production environments, which I'm eager to achieve, I will write about it too!