Samba CI on gitlab/Under the hood: Difference between revisions

From SambaWiki
(Created page with "=How GitLab CI works at a High Level= ==Running remote scripts, displaying the output== Like the Samba build farm of old, GitLab CI is best seen as a system for running scri...")
 
(explain more clearly that gitlab CI uses docker specifically)
(19 intermediate revisions by 3 users not shown)
Line 1: Line 1:
=How GitLab CI works at a High Level=
=How GitLab CI works in Samba=


==Running remote scripts, displaying the output==
==Running remote scripts, displaying the output==


Like the Samba build farm of old, GitLab CI is best seen as a system for running scripts on remote hosts against a git checkout.
Like the Samba [https://git.samba.org/samba.git/?p=build-farm.git;a=blob;f=README build farm of old], GitLab CI is a system '''for running scripts on remote hosts''' against a git checkout.


===Pipelines===
In Samba's case, the remote script is '''[[autobuild.py]]''' plus some housekeeping before and after. The details is recorded in the '''.gitlab-ci-private'''


Samba uses a feature called [https://docs.gitlab.com/ee/ci/pipelines.html GitLab Pipelines] to orchestrate our CI.
===.gitlab-ci-private.yml vs .gitlab-ci.yml


===In-repo configuration===
We have two different CI configurations, one using the default name (so picked up by default by forks of our repo) and one that we specify in the [[Samba CI on gitlab#Where is the Samba CI repo on GitLab?|Common development repo]]


In Samba's case, the remote script is '''[[autobuild|script/autobuild.py]]''' plus some housekeeping before and after. The details is recorded in the '''.gitlab-ci*.yml''' files in the Samba tree (so it is maintained with the code).
==Wrapping docker==


* See also [https://docs.gitlab.com/ee/ci/quick_start/README.html|an introduction to setting up GitLab CI]
To get a consistent build environment docker images are used, so the scripts runs in a docker container. The image used is defined in the .gitlab-ci.yml file.

===.gitlab-ci-private.yml vs .gitlab-ci.yml===

We have two different CI configurations, one using the default name ''.gitlab-ci.yml'' (so picked up by default by forks of our repo) and one that we specify in the [[Samba CI on gitlab#Where is the Samba CI repo on GitLab?|Common development repo]] (''.gitlab-ci-private.yml'')

The ''.gitlab-ci-private.yml'' file includes ''.gitlab-ci.yml'' to as to avoid duplication.

The motivation here is to use the [https://about.gitlab.com/2016/04/05/shared-runners/ shared runners] where possible as these are provided by gitlab.com at no cost to Samba Team.

==Wrapping containers==

To get a consistent build environment container images are used, so the scripts described above all run inside a container.

The image used is defined in the .gitlab-ci.yml file.

'''GitLab CI is best thought of as a fancy way to run commands in containers and report their results.'''

===Docker===

GitLab CI uses [https://www.docker.com/ Docker] as the container runtime.

''While the container image format can be consumed by and the containers started using other tools, to closely replicate the environment on the runners, use Docker.
''
===A bit like running in a chroot===

The way containers are used by GitLab CI is very much akin to downloading a tarball (the image), unpacking it and calling ''chroot'' into it (entering the container). Modern container concepts like namespaces etc are used to make it more seamless, but this conceptualization may assist those struggling with the concepts.


==On a private VM==
==On a private VM==


To allow us to accept and test code from a broader range of contributors, and to enable scaling at times of peak load, the docker container is started in a private VM using [https://docs.docker.com/machine/overview/ Docker Machine]]. This applies for '''both''' the private and shared (provided by gitlab.com) runners.
To allow us to accept and test code from a broader range of contributors, and to enable scaling at times of peak load, the docker container is started in a private VM using [https://docs.docker.com/machine/overview/ Docker Machine]. This applies for '''both''' the private and shared (provided by gitlab.com) runners.

==Multiple VMs in parallel==

Each section in the '''.gitlab-ci*.yml''' file is a [https://docs.gitlab.com/ee/ci/yaml job], and each job is distributed to an independent VM, allowing execution in parallel.

=Providing the private VMs=

* The Samba team provides the ''private'' VMs using credit in the [https://www.rackspace.com/cloud/public Rackspace cloud].

* A single host running [https://docs.gitlab.com/runner/install/ gitlab-runner] is registered to the [[Samba CI on gitlab#Where is the Samba CI repo on GitLab?|shared development repo]].

* That host is configured to [https://docs.gitlab.com/runner/configuration/autoscale.html autoscale] [https://docs.gitlab.com/runner/executors/docker_machine.html using docker-machine].

=Ansible management scripts=

* The [https://gitlab.com/catalyst-samba/samba-cloud-autobuild/tree/master/gitlab-ci scripts used to configure and operate this service] are available.

=Future CI services=

As all the complex parts of Samba's build and test system are still below [[autobuild]], migration to a different CI service in the future or in parallel is quite practical.

For example, in the past there was parallel operation with [https://travis-ci.org/ Travis CI] before the team abandoned [https://github.com/samba-team/samba GitHub].

==Not tied to gitlab.com==

If needed, private GitLab hosts running the Open Source GitLab CE can interpret the same configuration and operate against the same runners (just without the free shared runners, naturally).

This gives the Samba Team options if gitlab.com hosting becomes a problem for any reason.

Revision as of 21:36, 18 February 2019

How GitLab CI works in Samba

Running remote scripts, displaying the output

Like the Samba build farm of old, GitLab CI is a system for running scripts on remote hosts against a git checkout.

Pipelines

Samba uses a feature called GitLab Pipelines to orchestrate our CI.

In-repo configuration

In Samba's case, the remote script is script/autobuild.py plus some housekeeping before and after. The details is recorded in the .gitlab-ci*.yml files in the Samba tree (so it is maintained with the code).

.gitlab-ci-private.yml vs .gitlab-ci.yml

We have two different CI configurations, one using the default name .gitlab-ci.yml (so picked up by default by forks of our repo) and one that we specify in the Common development repo (.gitlab-ci-private.yml)

The .gitlab-ci-private.yml file includes .gitlab-ci.yml to as to avoid duplication.

The motivation here is to use the shared runners where possible as these are provided by gitlab.com at no cost to Samba Team.

Wrapping containers

To get a consistent build environment container images are used, so the scripts described above all run inside a container.

The image used is defined in the .gitlab-ci.yml file.

GitLab CI is best thought of as a fancy way to run commands in containers and report their results.

Docker

GitLab CI uses Docker as the container runtime.

While the container image format can be consumed by and the containers started using other tools, to closely replicate the environment on the runners, use Docker.

A bit like running in a chroot

The way containers are used by GitLab CI is very much akin to downloading a tarball (the image), unpacking it and calling chroot into it (entering the container). Modern container concepts like namespaces etc are used to make it more seamless, but this conceptualization may assist those struggling with the concepts.

On a private VM

To allow us to accept and test code from a broader range of contributors, and to enable scaling at times of peak load, the docker container is started in a private VM using Docker Machine. This applies for both the private and shared (provided by gitlab.com) runners.

Multiple VMs in parallel

Each section in the .gitlab-ci*.yml file is a job, and each job is distributed to an independent VM, allowing execution in parallel.

Providing the private VMs

  • The Samba team provides the private VMs using credit in the Rackspace cloud.

Ansible management scripts

Future CI services

As all the complex parts of Samba's build and test system are still below autobuild, migration to a different CI service in the future or in parallel is quite practical.

For example, in the past there was parallel operation with Travis CI before the team abandoned GitHub.

Not tied to gitlab.com

If needed, private GitLab hosts running the Open Source GitLab CE can interpret the same configuration and operate against the same runners (just without the free shared runners, naturally).

This gives the Samba Team options if gitlab.com hosting becomes a problem for any reason.