Using git bisect to locate a Samba issue

Background

You may discover a bug or regression in one version of samba, that is not present in another version.

The "git bisect" tool is a good way to automate the search for the specific commit that changed the behaviour, however you need to provide the tool with an automated way of testing for the condition, so that 'git bisect' can determine whether a particular samba version is good or bad.

The example shown on this page is a LDAP permission issue that needs a running samba domain controller to replicate. Therefore, the test script shown below uses docker to build, install and run each version of samba in a docker container and then use 'ldapsearch' to check the returned search results against the expected values. You will of course need to adjust this test script to check for the specific issue you are experiencing.

To create test data from your domain, use the Domain rename tool to create a backup; the example on this page assumes you have used mydomain.org as the renamed domain, e.g.:

sudo samba-tool domain backup rename MYDOMAIN mydomain.org --targetdir=/tmp/backup --server=dc1 -UAdministrator

Pre-requisites

Using the Dockerfile below, create a Docker image named "sambabuildenv" that provides an environment suitable for building and running samba:

docker -t sambabuildenv .

Then, obtain the Samba source code using git - this will be used by the 'git bisect' tool to search the previous commits for where the issue was introduced. This guide assumes you are working in the "~/docker/sambatest" directory; change this path as required for your environment.

cd ~/docker/sambatest
git clone https://gitlab.com/samba-team/samba.git

Finally, create "bisect-test.sh" from the example below. This will be used by git bisect to determine if each version is 'good' or 'bad', and will need to be customised to your own requirements. The example shows how an LDAP search can be run; if your example does not need a running samba DC then this script can be much simplified and the "Installing, restoring and starting" step may not be needed.

Running the git bisect itself

Specify the known bad and known good versions or commit IDs to search between. In the example below, 4.18.5 is known to have a bug, and 4.11.18 is known to work OK.

cd ~/docker/sambatest/samba
git bisect start samba-4.18.5 samba-4.11.18

Then start the bisect itself

git bisect run ~/docker/sambatest/bisect-test.sh

This process will run for some time, showing its progress and roughly how many revisions/steps remain to be tested. At any point you can use a different terminal window and check what steps have been taken so far in the bisect:

cd ~/docker/sambatest/samba   # change to the git code directory
git bisect log

Eventually, the output of your 'git bisect run' command will show you the first bad commit, i.e. the code that was introduced that changed Samba's behaviour according to your test criteria.

File contents

The contents of these files are user-generated and have not been created by the Samba team; therefore they may well not be optimised, efficient or the best way to do this. However they have been successfully used to identify a live regression and it is felt that this example will be useful to help others who may not have used 'git bisect' before. Caveat Emptor!

Dockerfile

FROM debian:bullseye-slim

RUN DEBIAN_FRONTEND=noninteractive apt update

RUN DEBIAN_FRONTEND=noninteractive apt -y install acl attr autoconf bison build-essential debhelper dnsutils docbook-xml docbook-xsl flex gdb krb5-user libacl1-dev libaio-dev libattr1-dev libblkid-dev libbsd-dev libcap-dev libcups2-dev libgnutls28-dev libjson-perl libldap2-dev libncurses5-dev libpam0g-dev libparse-yapp-perl libpopt-dev libreadline-dev perl perl-modules pkg-config python-all-dev python-dev xsltproc zlib1g-dev libarchive-dev net-tools python3-markdown liblmdb-dev wget python3-dev libjansson-dev python3-dnspython ldap-utils

bisect-test.sh

#!/bin/bash

# 'git bisect' will run this script for each revision it tests.
# The job of this script is to compile & launch Samba, run the necessary check,
# and return an exit code of 0 for OK or 1 for failure.

# In this example, this script restores data from a domain backup and then
# runs samba locally for a ldapsearch query

# Define cleanup function for before we exit
function docker_cleanup () {
	# Preserve docker logs in our log folder
	docker logs sambarestoretest > "${LOGFILE_DOCKERRUN}"

	( echo -n "Stopping: "; docker stop sambarestoretest ) >>"${LOGFILE}" 2>&1;
	( echo -n "Removing: "; docker rm   sambarestoretest ) >>"${LOGFILE}" 2>&1;
}

BACKUPFILE=samba-backup-mydomain.org-2023-11-07T19-59-24.931064.tar.bz2

THISVERSION=$(git describe --tags)
if [ "$1" == "" ]; then
	LOGDIR=/home/user/docker/sambatest/logs
else
	LOGDIR=$1
fi
# Record date and time once, then refer to it below, to avoid complications if running over midnight
TIMESTAMP=$(date "+%H%M%S")
DATESTAMP=$(date "+%Y%m%d")
LOGFILE=${LOGDIR}/${DATESTAMP}-${TIMESTAMP}-${THISVERSION}.txt
LOGFILE_DOCKERRUN=${LOGDIR}/${DATESTAMP}-${TIMESTAMP}-${THISVERSION}-docker_run.txt
LOGFILE_SUMMARY=${LOGDIR}/${DATESTAMP}-summary.txt
echo "===> Logging to ${LOGFILE}"
echo "===> Docker run log to ${LOGFILE_DOCKERRUN}"
echo "===> Summary to ${LOGFILE_SUMMARY}"

# Compile this version of samba
echo "===> Compiling tag '${THISVERSION}' at ${TIMESTAMP}" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}"
docker run -it --rm -v ./:/build sambabuildenv /bin/bash -c "unset TERM; cd /build; make distclean; make clean ; ./configure --without-gpgme --with-shared-modules='!vfs_snapper'; make -j 7" >>"${LOGFILE}" 2>&1

cd ..

# Did the compilation complete successfully?
# (ldbsearch isn't used in the next step but it's a useful indicator of success)
if [ ! -e samba/bin/default/lib/ldb/ldbsearch ]; then
	echo "ldbsearch not found - compile failed?" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}"
	# Return an exit code of over 127 to abort and quit
	# Return an exit code of exactly 125 to skip current version and continue (after resetting tree)
	cd - || exit 200	# panic if previous directory no longer exists
	git clean -f -d
	exit 125
fi

# Install samba, restore the backup, and start smbd
# SYS_ADMIN capability in docker is required for NTACL as part of domain restore
# Can also use --no-secrets on the domain restore, to not recover password hashes,
# depending on what is being tested for
echo "===> Installing, restoring and starting" | tee -a "${LOGFILE}"
docker run -d -it --name sambarestoretest \
	-v ./samba/:/build \
	-v ./smb.conf:/usr/local/samba/etc/smb.conf \
	-v ./${BACKUPFILE}:/backupfile:ro \
	--cap-add=SYS_ADMIN \
	sambabuildenv \
	/bin/bash -c "unset TERM; echo '==> Installing'; cd /build; make install && echo '==> Restoring' && /usr/local/samba/bin/samba-tool domain backup restore --newservername=testdc1 --targetdir=/usr/local/sambarestore --backup-file=/backupfile && /usr/local/samba/bin/samba-tool user setpassword Administrator -s /usr/local/sambarestore/etc/smb.conf --option='check password script'='' --newpassword='Debugging99' && echo '==> Starting' && /usr/local/samba/sbin/samba -s /usr/local/sambarestore/etc/smb.conf -F -i -d 2" >>"$LOGFILE" 2>&1

# Wait for the DC to start up
echo "===> Waiting for samba to install, restore and start" | tee -a "${LOGFILE}"
GREPOUTPUT=$(docker logs -f sambarestoretest | grep -E -m 1 "^samba version 4.*started")
GREPRET=$?
echo "$GREPOUTPUT" | tee -a "${LOGFILE}"; unset GREPOUTPUT

if [ "${GREPRET}" == "1" ]; then
	# docker logs exited, rather than grep succeeding
	echo samba did not start, exiting. | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}"
    echo Last few lines of docker log follows | tee -a "${LOGFILE}"
	( docker logs sambarestoretest | tail ) >>"$LOGFILE" 2>&1

	# XXX Debug: Allow more investigation as to why samba did not start
	#read -p "DEBUG: Press ENTER when done..." >&2

	docker_cleanup
	# Exit with return code 125 to skip this revision
	# Samba not starting isn't the bug we're looking for
	# An example could be build errors or e.g. https://bugzilla.samba.org/show_bug.cgi?id=14209
	cd - || exit 200	# panic if previous directory no longer exists
	git clean -f -d
	exit 125
fi
# And then another small delay to be sure it has started
sleep 5

# Run the test itself
echo "===> Obtaining test results" | tee -a "${LOGFILE}"
EXPECTEDRESULTS=43
CMDOUTPUT=$(
docker run -it --rm -v ./samba/:/build \
	--link sambarestoretest:dc1.mydomain.org \
	-e LDAPTLS_REQCERT=never \
	sambabuildenv \
	ldapsearch -H ldap://dc1.mydomain.org -ZZ -x -w Debugging99 -D Administrator@mydomain \
	-b "dc=mydomain,dc=org" \
	"(&(objectCategory=Person)(sAMAccountName=*)(memberOf:1.2.840.113556.1.4.1941:=CN=MyGroup,OU=MyOU,DC=mydomain,DC=org))" samAccountName 2>>"${LOGFILE}"
)
# Placing the variable reference within double quotes will print embedded newlines
SEARCHRESULTS=$(echo "$CMDOUTPUT" | grep numResponses | cut -f 3 -d ' ' | tr -d '\n\r')
echo "==> Got '${SEARCHRESULTS}'" | tee -a "${LOGFILE}"
# XXX Debug: Allow more investigation if needed
#read -p "Press ENTER - chance to debug using the running container"

echo "===> Cleaning up" | tee -a "${LOGFILE}"
# Remove our temporary docker image
docker_cleanup

# Clean up so as to ensure the next run doesn't start if we don't have a succesful compilation
#docker run -it --rm -v ./:/build sambabuildenv rm /build/bin/ldbsearch /build/bin/default/lib/ldb/ldbsearch 2>&1 >>$LOGFILE
# Also clean up so that 'git bisect' can checkout the next branch without complaints of files left behind
cd - || exit 200	# panic if previous directory no longer exists
git clean -f -d

echo "===> Checking test results" | tee -a "${LOGFILE}"
# Check the returned result and exit
if [ "z${SEARCHRESULTS}" == "z${EXPECTEDRESULTS}" ]; then
	echo "Success - got '${SEARCHRESULTS}'" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}"
	exit 0
else
	echo "Failure - got '${SEARCHRESULTS}'" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}"
	echo "Command output was: '${CMDOUTPUT}'" >> "${LOGFILE}"
	exit 1
fi