Automation is one of the biggest levers IT brings naturally with it. Really starting to make use of its advantages is another chapter and today I would like to go through the process of how to build a really simple CI/CD Pipeline using popular tools like GitLab, Docker and Amazon Web Services. Above a rough overview of what we are going to focus on.
The aim should be to be able to push our code to a GitLab Repository, GitLab then builds a Docker container image and pushes it to a Container Registry on AWS.
1. Create GitLab Repository
Creating a new GitLab Repository using the “Create blank project” button. Choose a project name in the following step and press “create project”.
2. Initialize the GitLab Runner Instance
We want to use the AWS cloud services to handle the workloads and to store our Docker image(s) later on.
Firstly, we create an EC2 instance, which will do the main work when in comes to building the image and pushing it to our Registry. I named the EC2 instance “GitLab Runner”.
You can basically choose an instance type at your choice – I am going with a t2.small instance. Make also sure, that your Security Group settings for the EC2 instance allow respective ports for incoming requests (link). Otherwise GitLab will not be able to communicate with the instance.
Now, connect to the EC2 instance via SSH / Terminal. You should see something similar:
Run the command
>> sudo apt-get update
One of the most important elements for our CI/CD pipeline is the so called ‘GitLab Runner’, who takes care of the whole realization of our pipeline. It needs to be installed on our EC2 instance and needs to be registered in the GitLab Project. Moving to the Settings of your GitLab project and choosing CI/CD, you should see the following (make sure to expand the Runners section):
Click on ‘Show Runner Installation Instructions’ and follow the steps which are displayed (except the registration of the runner).
After setting up the GitLab-Runner, we have to register it to our GitLab project using the following command (replace the ‘REGISTRATION_TOKEN‘ with your own Token from the GitLab Project).
sudo gitlab-runner register -n \ --url https://gitlab.com/ \ --registration-token REGISTRATION_TOKEN \ --executor shell \ --description "GitLab Runner"
If everything went successfully, take a look into your GitLab Project > Settings > CI/CD > Runners – it should display a runner with a green dot on the left side. Now we have created our GitLab Runner and it will listen to each push we are going to make in the future to this GitLab repository.
After preparing the GitLab runner on our EC2 instance, we should now install Docker using the Terminal and follwing lines (https://docs.gitlab.com/ee/ci/docker/using_docker_build.html):
>> sudo apt update >> sudo apt install apt-transport-https ca-certificates curl software-properties-common >> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - >> sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable" >> sudo apt update >> apt-cache policy docker-ce >> sudo apt install docker-ce >> sudo systemctl status docker >> sudo usermod -aG docker gitlab-runner >> sudo -u gitlab-runner -H docker info
If the last command gives you a reasonable output, everything is fine and you can move on to the next step, which is basically installing the AWS Command Line Interface (CLI):
You can follow this article to install the CLI: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html
Now that you have installed the aws cli, we can configure the AWS user, so that we can use services of our AWS account. In order to get your secrets, go into your management console > security login information and create your secrets. Then, perform following lines and enter the respective information.:
>> sudo su gitlab-runner >> cd >> aws configure
Now your AWS account should be connected to the AWS CLI.
In case of an error LATER: In some cases, there is an error from the GitLab pipeline (which is also reported here). I could solve it by deleting the files in the /home/gitlab-runner directory:
>> cd /home/gitlab-runner >> sudo rm .*
3. Create Docker Image Repository
Open the Elastic Container Registry service in your AWS management console and click on ‘create repository’ (it does not matter if it is a private or public repository, it will work either way). For now, there is not much to do in the ECR Repository.
4. Create Project Files
Now we can start working on our real project files. Basically we need four files:
- application.py file (python file with program code)
- requirements.txt file (to install the packages we need for the application)
- Dockerfile (to create the docker container)
- .gitlab-ci.yml file (to configure the CI/CD pipeline)
### application.py print("HELLO") # Normally, here would be your complicated code...
### requirements.txt numpy==1.20.1
### Dockerfile FROM python:3.9 COPY . /appdir WORKDIR /appdir RUN pip install -r requirements.txt EXPOSE 5000 CMD ["python", "application.py"]
# .gitlab-ci.yml image: docker:latest services: - docker:dind stages: - build variables: TEST_IMAGE: REPOSITORY_LINK:TAG_NAME before_script: - docker info - aws --version - whoami build: stage: build script: - aws ecr-public get-login-password --region REGION | docker login --username AWS --password-stdin REPOSITORY_LINK - docker build -t IMAGE_NAME . - docker tag IMAGE_NAME:TAG_NAME $TEST_IMAGE - docker push $TEST_IMAGE
5. Push code
After we have created the above shown files, push them all together into the GitLab Repository we have created in the beginning.
Wait and see your GitLab Pipeline working (GitLab Project > CI/CD > Jobs)… and hopefully in a succesful way…
In case there are issues, just click on the job which failed and see the logs.
I hope you liked this tutorial – in case of questions don’t hesitate to ask me.