Understanding make test

Running tests

In samba development, one sure way to check if your code has not broken any existing functionality is to execute following command.

make test

This command tests various parts of samba code base to ensure the correctness of the functionality. It uses the test framework built in samba code base to perform hundreds of tests. It does take a fair bit of time to execute all the tests. If all the tests are successful, ALL OK is issued and the summary of the results can be found in st/summary file. In case any of the tests are unsuccessful, st/summary file should indicate whether the failures are and st/subunit file will capture all the output for all the tests.

What's quicktest

Often you can test the core functionality, by performing a quick test as follows.

make quicktest

This will perform few dozen tests to quickly check if anything is broken in the code.

How to run a particular test or a set of tests

Each test in samba has an unique name. To run a particular test, you can specify the test name to TESTS argument.

make test TESTS=samba3.blackbox.net.local.registry.roundtrip

The argument TESTS is really a regular expression. Any test name that matches the regular expression will be executed. To execute the same test, one can use the following command.

make test TESTS=roundtrip

Now, this will execute two tests - samba3.blackbox.net.local.registry.roundtrip and samba3.blackbox.net.rpc.registry.roundtrip.

How to find what tests are there?

If you are impatient and want to find what all tests are performed, use the following command.

make test LIST=1

To understand how this command really works, keep reading.

Make magic

make reads Makefile to build targets using specified rules. First few lines from Makefile in samba source shows the following.

# simple makefile wrapper to run waf

WAF_BINARY=./buildtools/bin/waf
WAF=WAF_MAKE=1 $(WAF_BINARY)

all:
        $(WAF) build

install:
        $(WAF) install

uninstall:
        $(WAF) uninstall

test:
        $(WAF) test $(TEST_OPTIONS)

A target test can built using specified rule '$(WAF) test $(TEST_OPTIONS)', which executes ./buildtools/bin/waf with test as an argument.


Some 'Waf'fling

Samba uses python based build system called waf. If you are only used to automake/make/cmake family for building applications, then waf can be a bit daunting.

To get familiar with waf, you can use following resources.

Just like you have Makefile for make, there is wscript for waf. A wscript file is essentially a python script file. And the build targets are defined as python functions. An excerpt from wscript in samba source is shown below.

#! /usr/bin/env python

srcdir = '.'
blddir = 'bin'

APPNAME='samba'
VERSION=None

import sys, os
sys.path.insert(0, srcdir+"/buildtools/wafsamba")
import wafsamba, Options, samba_dist, Scripting, Utils, samba_version


samba_dist.DIST_DIRS('.')

# install in /usr/local/samba by default
Options.default_prefix = '/usr/local/samba'

def set_options(opt):
    opt.BUILTIN_DEFAULT('NONE')
    opt.PRIVATE_EXTENSION_DEFAULT('samba4')
    opt.RECURSE('lib/replace')
    opt.RECURSE('dynconfig')
    opt.RECURSE('lib/ldb')
    opt.RECURSE('selftest')
    opt.RECURSE('source4/lib/tls')
    opt.RECURSE('lib/nss_wrapper')
    opt.RECURSE('lib/socket_wrapper')
    opt.RECURSE('lib/uid_wrapper')
    opt.RECURSE('pidl')
    opt.RECURSE('source3')

    gr = opt.option_group('developer options')
    gr.add_option('--enable-build-farm',
                   help='enable special build farm options',
                   action='store_true', dest='BUILD_FARM')

    opt.tool_options('python') # options for disabling pyc or pyo compilation
    # enable options related to building python extensions


  [...more python...]


def main():
    from samba_wildcard import wildcard_main
    wildcard_main(wildcard_cmd)


There is no python function test() in wscript corresponding to target test. Similarly there is no function build() as required by 'make all'. Now the things start getting complicated. Before waf tries to find any of the functions corresponding to the targets, it executes set_options() function.

(For the curious ones - The code is in buildtools/.waf-xxxxx/wafadmin/Scripting.py, function prepare_impl)

Samba has extended waf functionality to make it easier for developers to specify targets/dependencies and to add special magic to detect duplicate symbol linking. One of the extension is RECURSE() function (from buildtools/wafsamba/samba_utils.py) which looks inside the specified directory for wscript files and parses them. And the line of interest is opt.RECURSE('selftest'). This tells waf to parse selftest/wscript file, which has following code.

def set_options(opt):

    opt.add_option('--enable-selftest',
                   help=("enable options necessary for selftest (default=no)"),
                   action="store_true", dest='enable_selftest', default=False)
    opt.add_option('--with-selftest-prefix',
                   help=("specify location of selftest directory (default=./st)"),
                   action="store", dest='SELFTEST_PREFIX', default='./st')

    opt.ADD_COMMAND('test', cmd_test)
    opt.ADD_COMMAND('testonly', cmd_testonly)


  [...more python...]


# main test entry point
def cmd_test(opt):
    '''Run the test suite (see test options below)'''

    # if running all tests, then force a symbol check
    env = LOAD_ENVIRONMENT()
    CHECK_MAKEFLAGS(env)
    if not Options.options.TESTS:
        Options.options.DUP_SYMBOLCHECK = True

    Scripting.commands.append('build')
    Scripting.commands.append('testonly')

Digging Deeper