Businesses often strive to autoscale despite being on the cloud. There are many underlying issues that create hindrance in autoscaling.
And sometimes, it’s become difficult to get to the roots of such issues.
Exact same thing happened with one of our customers.
Like most businesses, 7Targets is also having their code repositories in GitLab and using GCP instances as a GitLab runner.
Continuous Integration and Continuous Deployment won’t work without a GitLab runner. GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.
But, due to some reasons, the runner instance’s availability zone was down. So, our customer was not able to push any changes because without a runner job won’t start.
I suggested the client create an autoscaled GitLab runner on AWS. So, in future, any of the cloud providers instance’s availability zone goes down, it won’t affect the work going on.
Below are the steps I followed to create an autoscaled GitLab CI/CD runner:
- The first step is to install GitLab Runner in the EC2 instance, which will act as the runner manager which spawns the new machines. Choose a distribution that supports both Docker and GitLab Runner, such as Amazon Linux, Ubuntu, Debian, CentOS, etc.
- You can choose any small instance type such as t2.micro as it will not run any tasks by itself. This machine will be a dedicated host as we always need it up, thus it will be the only standard cost.
- Install the prerequisites:
a) Connect to EC2 Instance:
b) Add official GitLab repository:
- For Debian/Ubuntu/Mint:
curl -L "https://packages.gitLab.com/install/repositories/runner/gitLab-runner/script.deb.sh" | sudo bash
- For RHEL/CentOS/Fedora:
curl -L "https://packages.gitLab.com/install/repositories/runner/gitLab-runner/script.rpm.sh" | sudo bash
Install GitLab Runner from the GitLab repository by entering the below cmds.
- For Debian/Ubuntu/Mint:
sudo apt-get install gitlab-runner
- For Debian/Ubuntu/Mint:
sudo yum install -y gitlab-runner
Want to know more? Let’s discuss.
c) Install Docker
- For Debian/Ubuntu/Mint:
sudo apt-get install docker
- For RHEL/CentOS/Fedora:
sudo yum install docker
d) Register the Runner
- Run the following command:
sudo gitlab-runner register
- Enter your GitLab instance URL.
- Enter the token received to register the runner. (As shown in below image, You will find GitLab Instance URL and registration token in Project Group Settings > CI/CD>Runners)
- Enter the details for the runner. You can change this value later in the GitLab UI (User Interface).
- Enter tag associated with runner, separated by comma. You can change this value later in the GitLab UI.
- Enter a maintenance note for the runner(optional).
- Enter the runner executor.
- If you enter docker or docker+machine as executor, you have to enter the default image to be used for the projects that don’t define the image in .GitLab-ci.yml.
4. Configuring the Runner
Now that runner is registered, you need to edit its configuration file and add the necessary options for the AWS machine driver.
Configuration File Path: /etc/gitlab-runner/config.toml
- In the global section, you can define the limit of jobs that can be run simultaneously across all runners (concurrent). This steadily depends on your needs, like how many users GitLab Runner will consider, how much time your builds take, etc. You can start with a low value like 10, and increase or decrease the value going forward.The check_interval option means how often the runner should check GitLab for new jobs (in seconds).
- From the [[runners]] section, the principal part is the executor which needs to be set to docker+machine. Most of those configurations are taken care of when you register the runner for the first time.
- In the [runners.docker] section you can specify the default Docker image used by child runners if it’s not specified in .GitLab-ci.yml. Using privileged = true, all runners will be able to run Docker in Docker which is useful if you want to build your own custom Docker images via GitLab CI/CD.
- Add the “/var/run/docker.sock:/var/run/docker.sock” to the array of volumes. Docker socket file is located at /var/run/docker.sock and it is used to communicate with the main docker daemon process by default. It is the entry point for a Docker API. By default, This socket is used by Docker CLI to execute docker commands.
Need more help? - [runners.machine] is the most important part of the configuration and it’s the one that tells GitLab Runner how and when to spawn new or remove old Docker Machine instances.
- Enter the Docker Machine driver, here it is set to amazonec2 and the machine name has a standard prefix followed by %s (required) that will be replaced by the ID of child runner: GitLab-docker-machine-%s.Now, based on your AWS infrastructure, there are a number of options that you can set up under MachineOptions. Here, you can see the most common options.
- In [[runners.machine.autoscaling]] section you can define the following:
- IdleCount means number of machines that need to be created and waiting in Idle state. At the beginning, when no jobs are queued, GitLab Runner starts two machines (If you give IdleCount = 2), and sets them in Idle state.
- If IdleTime is set to 0, the autoscaled runners are terminated just after finishing the job.
- IdleTime means Time (in seconds) for a machine to be in Idle state before it is removed. In case of IdleCount=60, wait for 60 seconds before actually terminating the runner.
In GitLab, You will get the runner details in Project Settings > CI/CD > Runners (As shown in the below image)
Autoscale with us
We at AAIC, help businesses with cloud and DevOps services to boost your time to market. With a team of AWS certified engineers and architects along with our DevOps experts, we support you to plan and implement a cloud strategy in line with an agile approach. With us, you can push your apps to the market faster.
Driven by automation, we help customers to improve their software pipeline and on-premise infrastructure with cloud and DevOps.
Want to know more? Book a free consultation call.
Published By
Manan Tank
Cloud Champion
Manan Tank is a passionate cloud expert. He is a fast learner and contributes to our major projects. In his free time, he loves to travel and do road trips.
