Using git bisect to locate a Samba issue: Difference between revisions
(Guide to using git bisect with samba) |
m (Update to match latest version in use here - adds summary log) |
||
(5 intermediate revisions by the same user not shown) | |||
Line 3: | Line 3: | ||
You may discover a bug or regression in one version of samba, that is not present in another version. |
You may discover a bug or regression in one version of samba, that is not present in another version. |
||
The [https://git-scm.com/docs/git-bisect |
The [https://git-scm.com/docs/git-bisect "git bisect"] tool is a good way to automate the search for the specific commit that changed the behaviour, however you need to provide the tool with an automated way of testing for the condition, so that 'git bisect' can determine whether a particular samba version is good or bad. |
||
The example shown on this page is a LDAP permission issue that needs a running samba domain controller to replicate. Therefore, the test script shown below uses docker to build, install and run each version of samba in a docker container and then use 'ldapsearch' to check the returned search results against the expected values. You will of course need to adjust this test script to check for the specific issue you are experiencing. |
The example shown on this page is a LDAP permission issue that needs a running samba domain controller to replicate. Therefore, the test script shown below uses docker to build, install and run each version of samba in a docker container and then use 'ldapsearch' to check the returned search results against the expected values. You will of course need to adjust this test script to check for the specific issue you are experiencing. |
||
Line 21: | Line 21: | ||
git clone https://gitlab.com/samba-team/samba.git |
git clone https://gitlab.com/samba-team/samba.git |
||
Finally, create "bisect-test.sh" from the example below. This will be used by git bisect to determine if each version is 'good' or 'bad', and will need to be customised to your own requirements. The example shows how an LDAP search can be run; if your |
Finally, create "bisect-test.sh" from the example below. This will be used by git bisect to determine if each version is 'good' or 'bad', and will need to be customised to your own requirements. The example shows how an LDAP search can be run; if your example does not need a running samba DC then this script can be much simplified and the "Installing, restoring and starting" step may not be needed. |
||
== Running the git bisect itself == |
== Running the git bisect itself == |
||
Line 39: | Line 39: | ||
git bisect log |
git bisect log |
||
Eventually, the output of your 'git bisect run' command will show you the first bad commit, i.e. the code that was introduced that changed Samba's behaviour according to your test criteria. |
|||
= File contents = |
= File contents = |
||
Line 62: | Line 62: | ||
<nowiki>#!/bin/bash |
<nowiki>#!/bin/bash |
||
# 'git bisect' will run this script for each revision it tests. |
|||
⚫ | |||
# The job of this script is to compile & launch Samba, run the necessary check, |
|||
# and return an exit code of 0 for OK or 1 for failure. |
|||
# In this example, this script restores data from a domain backup and then |
|||
⚫ | |||
# Define cleanup function for before we exit |
# Define cleanup function for before we exit |
||
function docker_cleanup () { |
function docker_cleanup () { |
||
(echo -n "Stopping: "; docker stop sambarestoretest) |
( echo -n "Stopping: "; docker stop sambarestoretest ) >>"${LOGFILE}" 2>&1; |
||
(echo -n "Removing: "; docker rm sambarestoretest) |
( echo -n "Removing: "; docker rm sambarestoretest ) >>"${LOGFILE}" 2>&1; |
||
} |
} |
||
Line 74: | Line 79: | ||
THISVERSION=$(git describe --tags) |
THISVERSION=$(git describe --tags) |
||
if [ "$1" == "" ]; then |
if [ "$1" == "" ]; then |
||
LOGDIR=/home/ |
LOGDIR=/home/user/docker/sambatest/logs |
||
else |
else |
||
LOGDIR=$1 |
LOGDIR=$1 |
||
fi |
fi |
||
# Record date and time once, then refer to it below, to avoid complications if running over midnight |
|||
⚫ | |||
TIMESTAMP=$(date "+%H%M%S") |
|||
⚫ | |||
DATESTAMP=$(date "+%Y%m%d") |
|||
⚫ | |||
⚫ | |||
⚫ | |||
LOGFILE_SUMMARY=${LOGDIR}/${DATESTAMP}-summary.txt |
|||
⚫ | |||
echo "===> Summary to ${LOGFILE_SUMMARY}" |
|||
# Compile this version of samba |
# Compile this version of samba |
||
echo "===> Compiling tag '${THISVERSION}'" | tee ${LOGFILE} |
echo "===> Compiling tag '${THISVERSION}' at ${TIMESTAMP}" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}" |
||
docker run -it --rm -v ./:/build sambabuildenv /bin/bash -c "unset TERM; cd /build; make distclean; make clean ; ./configure --without-gpgme --with-shared-modules='!vfs_snapper'; make -j 7" |
docker run -it --rm -v ./:/build sambabuildenv /bin/bash -c "unset TERM; cd /build; make distclean; make clean ; ./configure --without-gpgme --with-shared-modules='!vfs_snapper'; make -j 7" >>"${LOGFILE}" 2>&1 |
||
cd .. |
cd .. |
||
Line 91: | Line 101: | ||
# (ldbsearch isn't used in the next step but it's a useful indicator of success) |
# (ldbsearch isn't used in the next step but it's a useful indicator of success) |
||
if [ ! -e samba/bin/default/lib/ldb/ldbsearch ]; then |
if [ ! -e samba/bin/default/lib/ldb/ldbsearch ]; then |
||
echo "ldbsearch not found - compile failed?" | tee -a ${LOGFILE} |
echo "ldbsearch not found - compile failed?" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}" |
||
# Return an exit code of over 127 to abort and quit |
# Return an exit code of over 127 to abort and quit |
||
# Return an exit code of exactly 125 to skip current version and continue (after resetting tree) |
# Return an exit code of exactly 125 to skip current version and continue (after resetting tree) |
||
cd - || exit 200 # panic if previous directory no longer exists |
|||
#exit 200 |
|||
cd - |
|||
git clean -f -d |
git clean -f -d |
||
exit 125 |
exit 125 |
||
Line 102: | Line 111: | ||
# Install samba, restore the backup, and start smbd |
# Install samba, restore the backup, and start smbd |
||
# SYS_ADMIN capability in docker is required for NTACL as part of domain restore |
# SYS_ADMIN capability in docker is required for NTACL as part of domain restore |
||
# Can use --no-secrets on the domain restore, to not recover password hashes, |
# Can also use --no-secrets on the domain restore, to not recover password hashes, |
||
# on what is being tested for |
# depending on what is being tested for |
||
echo "===> Installing, restoring and starting" | tee -a ${LOGFILE} |
echo "===> Installing, restoring and starting" | tee -a "${LOGFILE}" |
||
docker run -d -it --name sambarestoretest \ |
docker run -d -it --name sambarestoretest \ |
||
-v ./samba/:/build \ |
-v ./samba/:/build \ |
||
Line 111: | Line 120: | ||
--cap-add=SYS_ADMIN \ |
--cap-add=SYS_ADMIN \ |
||
sambabuildenv \ |
sambabuildenv \ |
||
/bin/bash -c "unset TERM; echo '==> Installing'; cd /build; make install && echo '==> Restoring' && /usr/local/samba/bin/samba-tool domain backup restore --newservername=testdc1 --targetdir=/usr/local/sambarestore --backup-file=/backupfile && /usr/local/samba/bin/samba-tool user setpassword Administrator -s /usr/local/sambarestore/etc/smb.conf --option='check password script'='' --newpassword='Debugging99' && echo '==> Starting' && /usr/local/samba/sbin/samba -s /usr/local/sambarestore/etc/smb.conf -F -i -d 2" |
/bin/bash -c "unset TERM; echo '==> Installing'; cd /build; make install && echo '==> Restoring' && /usr/local/samba/bin/samba-tool domain backup restore --newservername=testdc1 --targetdir=/usr/local/sambarestore --backup-file=/backupfile && /usr/local/samba/bin/samba-tool user setpassword Administrator -s /usr/local/sambarestore/etc/smb.conf --option='check password script'='' --newpassword='Debugging99' && echo '==> Starting' && /usr/local/samba/sbin/samba -s /usr/local/sambarestore/etc/smb.conf -F -i -d 2" >>"$LOGFILE" 2>&1 |
||
# Wait for the DC to start up |
# Wait for the DC to start up |
||
echo "===> Waiting for samba to install, restore and start" | tee -a ${LOGFILE} |
echo "===> Waiting for samba to install, restore and start" | tee -a "${LOGFILE}" |
||
GREPOUTPUT=$(docker logs -f sambarestoretest | |
GREPOUTPUT=$(docker logs -f sambarestoretest | grep -E -m 1 "^samba version 4.*started") |
||
GREPRET=$? |
GREPRET=$? |
||
echo $GREPOUTPUT | tee -a ${LOGFILE}; unset |
echo "$GREPOUTPUT" | tee -a "${LOGFILE}"; unset GREPOUTPUT |
||
# Preserve docker logs in our log folder |
# Preserve docker logs in our log folder |
||
docker logs sambarestoretest > ${LOGFILE_DOCKERRUN} |
docker logs sambarestoretest > "${LOGFILE_DOCKERRUN}" |
||
if [ "${GREPRET}" == "1" ]; then |
if [ "${GREPRET}" == "1" ]; then |
||
# docker logs exited, rather than grep succeeding |
# docker logs exited, rather than grep succeeding |
||
echo samba did not start, exiting. |
echo samba did not start, exiting. | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}" |
||
echo Last few lines of docker log follows | tee -a "${LOGFILE}" |
|||
( docker logs sambarestoretest | tail ) >>"$LOGFILE" 2>&1 |
|||
# XXX Debug: Allow more investigation as to why samba did not start |
|||
#read -p "DEBUG: Press ENTER when done..." >&2 |
|||
docker_cleanup |
docker_cleanup |
||
Line 130: | Line 143: | ||
# Samba not starting isn't the bug we're looking for |
# Samba not starting isn't the bug we're looking for |
||
# An example could be build errors or e.g. https://bugzilla.samba.org/show_bug.cgi?id=14209 |
# An example could be build errors or e.g. https://bugzilla.samba.org/show_bug.cgi?id=14209 |
||
cd - || exit 200 # panic if previous directory no longer exists |
|||
git clean -f -d |
|||
exit 125 |
exit 125 |
||
fi |
fi |
||
Line 137: | Line 151: | ||
# Run the test itself |
# Run the test itself |
||
echo "===> Obtaining test results" | tee -a ${LOGFILE} |
echo "===> Obtaining test results" | tee -a "${LOGFILE}" |
||
EXPECTEDRESULTS=43 |
EXPECTEDRESULTS=43 |
||
CMDOUTPUT=$( |
CMDOUTPUT=$( |
||
Line 146: | Line 160: | ||
ldapsearch -H ldap://dc1.mydomain.org -ZZ -x -w Debugging99 -D Administrator@mydomain \ |
ldapsearch -H ldap://dc1.mydomain.org -ZZ -x -w Debugging99 -D Administrator@mydomain \ |
||
-b "dc=mydomain,dc=org" \ |
-b "dc=mydomain,dc=org" \ |
||
"(&(objectCategory=Person)(sAMAccountName=*)(memberOf:1.2.840.113556.1.4.1941:=CN=MyGroup,OU=MyOU,DC=mydomain,DC=org))" samAccountName 2>>${LOGFILE} |
"(&(objectCategory=Person)(sAMAccountName=*)(memberOf:1.2.840.113556.1.4.1941:=CN=MyGroup,OU=MyOU,DC=mydomain,DC=org))" samAccountName 2>>"${LOGFILE}" |
||
) |
) |
||
# Placing the variable reference within double quotes will print embedded newlines |
# Placing the variable reference within double quotes will print embedded newlines |
||
SEARCHRESULTS= |
SEARCHRESULTS=$(echo "$CMDOUTPUT" | grep numResponses | cut -f 3 -d ' ' | tr -d '\n\r') |
||
echo "==> Got '${SEARCHRESULTS}'" | tee -a ${LOGFILE} |
echo "==> Got '${SEARCHRESULTS}'" | tee -a "${LOGFILE}" |
||
# XXX Debug: Allow more investigation if needed |
|||
#read -p "Press ENTER - chance to debug using the running container" |
|||
echo "===> Cleaning up" | tee -a ${LOGFILE} |
echo "===> Cleaning up" | tee -a "${LOGFILE}" |
||
# Remove our temporary docker image |
# Remove our temporary docker image |
||
docker_cleanup |
docker_cleanup |
||
# Clean up so as to ensure the next run doesn't start if we don't have a succesful compilation |
# Clean up so as to ensure the next run doesn't start if we don't have a succesful compilation |
||
#docker run -it --rm -v ./:/build sambabuildenv rm /build/bin/ldbsearch /build/bin/default/lib/ldb/ldbsearch 2>&1 >>$LOGFILE |
|||
# Also clean up so that 'git bisect' can checkout the next branch without complaints of files left behind |
# Also clean up so that 'git bisect' can checkout the next branch without complaints of files left behind |
||
cd - || exit 200 # panic if previous directory no longer exists |
|||
git clean -f -d |
|||
echo "===> Checking test results" | tee -a ${LOGFILE} |
echo "===> Checking test results" | tee -a "${LOGFILE}" |
||
# Check the returned result and exit |
# Check the returned result and exit |
||
if [ "z${SEARCHRESULTS}" == "z${EXPECTEDRESULTS}" ]; then |
if [ "z${SEARCHRESULTS}" == "z${EXPECTEDRESULTS}" ]; then |
||
echo Success - got |
echo "Success - got '${SEARCHRESULTS}'" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}" |
||
exit 0 |
exit 0 |
||
else |
else |
||
echo Failure - got |
echo "Failure - got '${SEARCHRESULTS}'" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}" |
||
echo Command output was: |
echo "Command output was: '${CMDOUTPUT}'" >> "${LOGFILE}" |
||
exit 1 |
exit 1 |
||
⚫ | |||
fi |
|||
⚫ |
Latest revision as of 13:14, 10 July 2024
Background
You may discover a bug or regression in one version of samba, that is not present in another version.
The "git bisect" tool is a good way to automate the search for the specific commit that changed the behaviour, however you need to provide the tool with an automated way of testing for the condition, so that 'git bisect' can determine whether a particular samba version is good or bad.
The example shown on this page is a LDAP permission issue that needs a running samba domain controller to replicate. Therefore, the test script shown below uses docker to build, install and run each version of samba in a docker container and then use 'ldapsearch' to check the returned search results against the expected values. You will of course need to adjust this test script to check for the specific issue you are experiencing.
To create test data from your domain, use the Domain rename tool to create a backup; the example on this page assumes you have used mydomain.org as the renamed domain, e.g.:
sudo samba-tool domain backup rename MYDOMAIN mydomain.org --targetdir=/tmp/backup --server=dc1 -UAdministrator
Pre-requisites
Using the Dockerfile below, create a Docker image named "sambabuildenv" that provides an environment suitable for building and running samba:
docker -t sambabuildenv .
Then, obtain the Samba source code using git - this will be used by the 'git bisect' tool to search the previous commits for where the issue was introduced. This guide assumes you are working in the "~/docker/sambatest" directory; change this path as required for your environment.
cd ~/docker/sambatest git clone https://gitlab.com/samba-team/samba.git
Finally, create "bisect-test.sh" from the example below. This will be used by git bisect to determine if each version is 'good' or 'bad', and will need to be customised to your own requirements. The example shows how an LDAP search can be run; if your example does not need a running samba DC then this script can be much simplified and the "Installing, restoring and starting" step may not be needed.
Running the git bisect itself
Specify the known bad and known good versions or commit IDs to search between. In the example below, 4.18.5 is known to have a bug, and 4.11.18 is known to work OK.
cd ~/docker/sambatest/samba git bisect start samba-4.18.5 samba-4.11.18
Then start the bisect itself
git bisect run ~/docker/sambatest/bisect-test.sh
This process will run for some time, showing its progress and roughly how many revisions/steps remain to be tested. At any point you can use a different terminal window and check what steps have been taken so far in the bisect:
cd ~/docker/sambatest/samba # change to the git code directory git bisect log
Eventually, the output of your 'git bisect run' command will show you the first bad commit, i.e. the code that was introduced that changed Samba's behaviour according to your test criteria.
File contents
![]() | The contents of these files are user-generated and have not been created by the Samba team; therefore they may well not be optimised, efficient or the best way to do this. However they have been successfully used to identify a live regression and it is felt that this example will be useful to help others who may not have used 'git bisect' before. Caveat Emptor! |
Dockerfile
FROM debian:bullseye-slim RUN DEBIAN_FRONTEND=noninteractive apt update RUN DEBIAN_FRONTEND=noninteractive apt -y install acl attr autoconf bison build-essential debhelper dnsutils docbook-xml docbook-xsl flex gdb krb5-user libacl1-dev libaio-dev libattr1-dev libblkid-dev libbsd-dev libcap-dev libcups2-dev libgnutls28-dev libjson-perl libldap2-dev libncurses5-dev libpam0g-dev libparse-yapp-perl libpopt-dev libreadline-dev perl perl-modules pkg-config python-all-dev python-dev xsltproc zlib1g-dev libarchive-dev net-tools python3-markdown liblmdb-dev wget python3-dev libjansson-dev python3-dnspython ldap-utils
bisect-test.sh
#!/bin/bash # 'git bisect' will run this script for each revision it tests. # The job of this script is to compile & launch Samba, run the necessary check, # and return an exit code of 0 for OK or 1 for failure. # In this example, this script restores data from a domain backup and then # runs samba locally for a ldapsearch query # Define cleanup function for before we exit function docker_cleanup () { ( echo -n "Stopping: "; docker stop sambarestoretest ) >>"${LOGFILE}" 2>&1; ( echo -n "Removing: "; docker rm sambarestoretest ) >>"${LOGFILE}" 2>&1; } BACKUPFILE=samba-backup-mydomain.org-2023-11-07T19-59-24.931064.tar.bz2 THISVERSION=$(git describe --tags) if [ "$1" == "" ]; then LOGDIR=/home/user/docker/sambatest/logs else LOGDIR=$1 fi # Record date and time once, then refer to it below, to avoid complications if running over midnight TIMESTAMP=$(date "+%H%M%S") DATESTAMP=$(date "+%Y%m%d") LOGFILE=${LOGDIR}/${DATESTAMP}-${TIMESTAMP}-${THISVERSION}.txt LOGFILE_DOCKERRUN=${LOGDIR}/${DATESTAMP}-${TIMESTAMP}-${THISVERSION}-docker_run.txt LOGFILE_SUMMARY=${LOGDIR}/${DATESTAMP}-summary.txt echo "===> Logging to ${LOGFILE}" echo "===> Summary to ${LOGFILE_SUMMARY}" # Compile this version of samba echo "===> Compiling tag '${THISVERSION}' at ${TIMESTAMP}" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}" docker run -it --rm -v ./:/build sambabuildenv /bin/bash -c "unset TERM; cd /build; make distclean; make clean ; ./configure --without-gpgme --with-shared-modules='!vfs_snapper'; make -j 7" >>"${LOGFILE}" 2>&1 cd .. # Did the compilation complete successfully? # (ldbsearch isn't used in the next step but it's a useful indicator of success) if [ ! -e samba/bin/default/lib/ldb/ldbsearch ]; then echo "ldbsearch not found - compile failed?" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}" # Return an exit code of over 127 to abort and quit # Return an exit code of exactly 125 to skip current version and continue (after resetting tree) cd - || exit 200 # panic if previous directory no longer exists git clean -f -d exit 125 fi # Install samba, restore the backup, and start smbd # SYS_ADMIN capability in docker is required for NTACL as part of domain restore # Can also use --no-secrets on the domain restore, to not recover password hashes, # depending on what is being tested for echo "===> Installing, restoring and starting" | tee -a "${LOGFILE}" docker run -d -it --name sambarestoretest \ -v ./samba/:/build \ -v ./smb.conf:/usr/local/samba/etc/smb.conf \ -v ./${BACKUPFILE}:/backupfile:ro \ --cap-add=SYS_ADMIN \ sambabuildenv \ /bin/bash -c "unset TERM; echo '==> Installing'; cd /build; make install && echo '==> Restoring' && /usr/local/samba/bin/samba-tool domain backup restore --newservername=testdc1 --targetdir=/usr/local/sambarestore --backup-file=/backupfile && /usr/local/samba/bin/samba-tool user setpassword Administrator -s /usr/local/sambarestore/etc/smb.conf --option='check password script'='' --newpassword='Debugging99' && echo '==> Starting' && /usr/local/samba/sbin/samba -s /usr/local/sambarestore/etc/smb.conf -F -i -d 2" >>"$LOGFILE" 2>&1 # Wait for the DC to start up echo "===> Waiting for samba to install, restore and start" | tee -a "${LOGFILE}" GREPOUTPUT=$(docker logs -f sambarestoretest | grep -E -m 1 "^samba version 4.*started") GREPRET=$? echo "$GREPOUTPUT" | tee -a "${LOGFILE}"; unset GREPOUTPUT # Preserve docker logs in our log folder docker logs sambarestoretest > "${LOGFILE_DOCKERRUN}" if [ "${GREPRET}" == "1" ]; then # docker logs exited, rather than grep succeeding echo samba did not start, exiting. | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}" echo Last few lines of docker log follows | tee -a "${LOGFILE}" ( docker logs sambarestoretest | tail ) >>"$LOGFILE" 2>&1 # XXX Debug: Allow more investigation as to why samba did not start #read -p "DEBUG: Press ENTER when done..." >&2 docker_cleanup # Exit with return code 125 to skip this revision # Samba not starting isn't the bug we're looking for # An example could be build errors or e.g. https://bugzilla.samba.org/show_bug.cgi?id=14209 cd - || exit 200 # panic if previous directory no longer exists git clean -f -d exit 125 fi # And then another small delay to be sure it has started sleep 5 # Run the test itself echo "===> Obtaining test results" | tee -a "${LOGFILE}" EXPECTEDRESULTS=43 CMDOUTPUT=$( docker run -it --rm -v ./samba/:/build \ --link sambarestoretest:dc1.mydomain.org \ -e LDAPTLS_REQCERT=never \ sambabuildenv \ ldapsearch -H ldap://dc1.mydomain.org -ZZ -x -w Debugging99 -D Administrator@mydomain \ -b "dc=mydomain,dc=org" \ "(&(objectCategory=Person)(sAMAccountName=*)(memberOf:1.2.840.113556.1.4.1941:=CN=MyGroup,OU=MyOU,DC=mydomain,DC=org))" samAccountName 2>>"${LOGFILE}" ) # Placing the variable reference within double quotes will print embedded newlines SEARCHRESULTS=$(echo "$CMDOUTPUT" | grep numResponses | cut -f 3 -d ' ' | tr -d '\n\r') echo "==> Got '${SEARCHRESULTS}'" | tee -a "${LOGFILE}" # XXX Debug: Allow more investigation if needed #read -p "Press ENTER - chance to debug using the running container" echo "===> Cleaning up" | tee -a "${LOGFILE}" # Remove our temporary docker image docker_cleanup # Clean up so as to ensure the next run doesn't start if we don't have a succesful compilation #docker run -it --rm -v ./:/build sambabuildenv rm /build/bin/ldbsearch /build/bin/default/lib/ldb/ldbsearch 2>&1 >>$LOGFILE # Also clean up so that 'git bisect' can checkout the next branch without complaints of files left behind cd - || exit 200 # panic if previous directory no longer exists git clean -f -d echo "===> Checking test results" | tee -a "${LOGFILE}" # Check the returned result and exit if [ "z${SEARCHRESULTS}" == "z${EXPECTEDRESULTS}" ]; then echo "Success - got '${SEARCHRESULTS}'" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}" exit 0 else echo "Failure - got '${SEARCHRESULTS}'" | tee -a "${LOGFILE}" "${LOGFILE_SUMMARY}" echo "Command output was: '${CMDOUTPUT}'" >> "${LOGFILE}" exit 1 fi