Samba's Continuous Integration (CI) process is called Autobuild. autobuild.py is a python script that verifies Samba builds, installs, and passes all automated tests successfully. Autobuild is generally invoked for CI automatically (e.g. via git hooks) rather than run manually.
Autobuild is more than just a wrapper for 'make test'. It runs additional tests that make test doesn't cover, such as the CTDB tests. Autobuild also builds various parts of the Samba project in different ways, for example using the LDB system library as well building the in-tree LDB source code. Running tests against these different build permutations ensures the main supported Samba configurations always work.
A successful autobuild is a gating step in delivering code to the master Samba branch. Autobuild is run on a host called sn-devel that only Samba Team members have access to. The script automatically rebases the commits under test to be on current master, so that if another change is merged into master first, testing restarts. Therefore, the Samba master branch always passes all tests.
Note that because autobuild is the gating step in code delivery, Samba Team members have the habit of (somewhat confusingly) using the term 'autobuild' to refer to delivering code, e.g. "I've pushed your code to autobuild".
Autobuild on sn-devel to push code (eg to master)
Samba team members have a git alias git autobuild configured that pushes the current branch to a magic location on the sn-devel server. If the scheduled run of script/autobuild.py passes on that server, the code will be marked and pushed to master via the --pushto and --mark parameters.
Bold text: the actual process is a bit more complex than this
Methods of running Autobuild
While autobuild.py can run on any Linux machine, the official results are only accepted on the sn-devel host. However, only Samba Team members can access sn-devel, and it's only a single host with limited CPU resources. So there are other methods that allow developers to get CI results in a similar way:
- Gitlab. Samba CI on gitlab is the preferred way for developers to run autobuild. Some tests can run against public 'runners', which gitlab provides for free. However, other tests need a specially configured CI box, due to memory or ext4 filesystem requirements. .gitlab-ci.yml implements Samba CI on Gitlab.
- Manually. Autobuild can also be run manually. A typical invocation of autobuild is:
- This can be run on any Linux system with the right packages installed. The current reference system has recently been switched over to Ubuntu 18.04 (for a long time it was Ubuntu 14.04).
- Openstack. autobuild.py should operate on a virtual machine in any OpenStack cloud.
Autobuild tests everything in the Samba codebase. However, the autobuild.py script defines a series of tasks, which allows many tests to be run in parallel. Because of this parallelization, a gitlab CI run usually takes about an hour to complete.
Potential improvements to Autobuild
Autobuild tests everything in the Samba codebase, which means it takes several hours to run. The autobuild.py script defines a series of independent tasks, which means some tests can be run in parallel, but only the cloud-based runners behind [Samba on GitLab|GitLab CI] fully takes advantage of that, as
sn-devel runs on one physical server.
The other problem is sometimes tests fail for unknown reasons at random times, i.e. the tests ‘flap’. A bot runs on
sn-devel four times per day and e-mails the samba-cvs mailing-list if any failures are detected. This allows developers to see which tests, perhaps newly introduced, caused the failure.
Tests can be marked as ‘flapping’ (either in selftest/flapping or selftest/flapping.d), which means the test case still gets run, but the test result is essentially ignored. Obviously, this practice is discouraged as it reduces the usefulness of the test.
More work could be done to investigate these intermittent test failures and fix them.