Understanding make test: Difference between revisions

From SambaWiki
(Created page with "== Introduction == In samba development, one sure way to check if your code has not broken any existing functionality is to execute following command. <pre> make test </pre> Th…")
 
(add info on the place of 'make test' in the world.)
 
(2 intermediate revisions by one other user not shown)
Line 1: Line 1:
'''make test''' is part of [[The Samba Selftest System]] and in turn [[Autobuild]] and [[Samba_on_GitLab|Gitlab CI]]
== Introduction ==

== Running tests ==
In samba development, one sure way to check if your code has not broken any existing functionality is to execute following command.
In samba development, one sure way to check if your code has not broken any existing functionality is to execute following command.


Line 17: Line 19:
This will perform few dozen tests to quickly check if anything is broken in the code.
This will perform few dozen tests to quickly check if anything is broken in the code.


=== How to find what tests are there? ===
=== How to run a particular test or a set of tests ===
Each test in samba has an unique name. To run a particular test, you can specify the test name to TESTS argument.
If you are impatient and want to find what all tests are performed, use the following command.


<pre>
<pre>
make test TESTS=samba3.blackbox.net.local.registry.roundtrip
make test LIST=1
</pre>
</pre>


The argument TESTS is really a ''regular expression''. Any test name that matches the regular expression will be executed. To
To understand how this command really works, keep reading.
execute the same test, one can use the following command.


<pre>
=== Some 'Waf'fling ===
make test TESTS=roundtrip
Samba uses python based build system called '''waf'''. If you are only used to automake/make/cmake family for building applications, then waf can be a bit daunting. Just like you have ''Makefile'' for make, there are ''wscript'' files for waf. A wscript file is really a python script file. And the ''build targets'' are defined as python functions.
</pre>


Now, this will execute two tests - '''samba3.blackbox.net.local.registry.roundtrip''' and '''samba3.blackbox.net.rpc.registry.roundtrip'''.
To get familiar with waf, you can check these links.
* [http://docs.waf.googlecode.com/git/book_16/single.html WAF book]
* [http://code.google.com/p/waf/ WAF website]


=== How to find what tests are there? ===
Samba has extended waf to wafsamba, to make it easier for developers to specify targets and their dependencies.
If you are impatient and want to find what all tests are performed, use the following command.


<pre>
==== Why waf? ====
make test LIST=1
*
</pre>

To understand how this command really works, keep reading.


=== Make magic ===
=== Make magic ===
make reads Makefile to build targets using specified rules. First few lines from Makefile in samba source shows the following.
The waf build system is wrapped at the top level inside a Makefile.


<pre>
<pre>
Line 60: Line 66:
</pre>
</pre>


make test really executes waf program as ''./buildtools/bin/waf'' with test as an argument.
A target ''test'' can built using specified rule '$(WAF) test $(TEST_OPTIONS)', which executes ''./buildtools/bin/waf'' with test as an argument.


== Some 'Waf'fling ==
Samba uses python based build system called '''waf'''. If you are only used to automake/make/cmake family for building applications, then waf can be a bit daunting.

To get familiar with waf, you can use following resources.
* [http://docs.waf.googlecode.com/git/book_16/single.html WAF book]
* [http://code.google.com/p/waf/ WAF website]

Just like you have ''Makefile'' for make, there is ''wscript'' for waf. A wscript file is essentially a python script file. And the ''build targets'' are defined as python functions. An excerpt from wscript in samba source is shown below.

<pre>
#! /usr/bin/env python

srcdir = '.'
blddir = 'bin'

APPNAME='samba'
VERSION=None

import sys, os
sys.path.insert(0, srcdir+"/buildtools/wafsamba")
import wafsamba, Options, samba_dist, Scripting, Utils, samba_version


samba_dist.DIST_DIRS('.')

# install in /usr/local/samba by default
Options.default_prefix = '/usr/local/samba'

def set_options(opt):
opt.BUILTIN_DEFAULT('NONE')
opt.PRIVATE_EXTENSION_DEFAULT('samba4')
opt.RECURSE('lib/replace')
opt.RECURSE('dynconfig')
opt.RECURSE('lib/ldb')
opt.RECURSE('selftest')
opt.RECURSE('source4/lib/tls')
opt.RECURSE('lib/nss_wrapper')
opt.RECURSE('lib/socket_wrapper')
opt.RECURSE('lib/uid_wrapper')
opt.RECURSE('pidl')
opt.RECURSE('source3')

gr = opt.option_group('developer options')
gr.add_option('--enable-build-farm',
help='enable special build farm options',
action='store_true', dest='BUILD_FARM')

opt.tool_options('python') # options for disabling pyc or pyo compilation
# enable options related to building python extensions


[...more python...]


def main():
from samba_wildcard import wildcard_main
wildcard_main(wildcard_cmd)


</pre>

There is no python function ''test()'' in wscript corresponding to target test. Similarly there is no function ''build()'' as required by 'make all'. Now the things start getting complicated. Before waf tries to find any of the functions corresponding to the targets, it executes '''set_options()''' function.

(For the curious ones - The code is in buildtools/.waf-xxxxx/wafadmin/Scripting.py, function prepare_impl)

Samba has extended waf functionality to make it easier for developers to specify targets/dependencies and to add special magic to detect duplicate symbol linking. One of the extension is RECURSE() function (from buildtools/wafsamba/samba_utils.py) which looks inside the specified directory for ''wscript'' files and parses them. And the line of interest is opt.RECURSE('selftest'). This tells waf to parse selftest/wscript file, which has following code.

<pre>
def set_options(opt):

opt.add_option('--enable-selftest',
help=("enable options necessary for selftest (default=no)"),
action="store_true", dest='enable_selftest', default=False)
opt.add_option('--with-selftest-prefix',
help=("specify location of selftest directory (default=./st)"),
action="store", dest='SELFTEST_PREFIX', default='./st')

opt.ADD_COMMAND('test', cmd_test)
opt.ADD_COMMAND('testonly', cmd_testonly)


[...more python...]


# main test entry point
def cmd_test(opt):
'''Run the test suite (see test options below)'''

# if running all tests, then force a symbol check
env = LOAD_ENVIRONMENT()
CHECK_MAKEFLAGS(env)
if not Options.options.TESTS:
Options.options.DUP_SYMBOLCHECK = True

Scripting.commands.append('build')
Scripting.commands.append('testonly')
</pre>


== Digging Deeper ==
== Digging Deeper ==

Latest revision as of 23:22, 4 June 2020

make test is part of The Samba Selftest System and in turn Autobuild and Gitlab CI

Running tests

In samba development, one sure way to check if your code has not broken any existing functionality is to execute following command.

make test

This command tests various parts of samba code base to ensure the correctness of the functionality. It uses the test framework built in samba code base to perform hundreds of tests. It does take a fair bit of time to execute all the tests. If all the tests are successful, ALL OK is issued and the summary of the results can be found in st/summary file. In case any of the tests are unsuccessful, st/summary file should indicate whether the failures are and st/subunit file will capture all the output for all the tests.

What's quicktest

Often you can test the core functionality, by performing a quick test as follows.

make quicktest

This will perform few dozen tests to quickly check if anything is broken in the code.

How to run a particular test or a set of tests

Each test in samba has an unique name. To run a particular test, you can specify the test name to TESTS argument.

make test TESTS=samba3.blackbox.net.local.registry.roundtrip

The argument TESTS is really a regular expression. Any test name that matches the regular expression will be executed. To execute the same test, one can use the following command.

make test TESTS=roundtrip

Now, this will execute two tests - samba3.blackbox.net.local.registry.roundtrip and samba3.blackbox.net.rpc.registry.roundtrip.

How to find what tests are there?

If you are impatient and want to find what all tests are performed, use the following command.

make test LIST=1

To understand how this command really works, keep reading.

Make magic

make reads Makefile to build targets using specified rules. First few lines from Makefile in samba source shows the following.

# simple makefile wrapper to run waf

WAF_BINARY=./buildtools/bin/waf
WAF=WAF_MAKE=1 $(WAF_BINARY)

all:
        $(WAF) build

install:
        $(WAF) install

uninstall:
        $(WAF) uninstall

test:
        $(WAF) test $(TEST_OPTIONS)

A target test can built using specified rule '$(WAF) test $(TEST_OPTIONS)', which executes ./buildtools/bin/waf with test as an argument.


Some 'Waf'fling

Samba uses python based build system called waf. If you are only used to automake/make/cmake family for building applications, then waf can be a bit daunting.

To get familiar with waf, you can use following resources.

Just like you have Makefile for make, there is wscript for waf. A wscript file is essentially a python script file. And the build targets are defined as python functions. An excerpt from wscript in samba source is shown below.

#! /usr/bin/env python

srcdir = '.'
blddir = 'bin'

APPNAME='samba'
VERSION=None

import sys, os
sys.path.insert(0, srcdir+"/buildtools/wafsamba")
import wafsamba, Options, samba_dist, Scripting, Utils, samba_version


samba_dist.DIST_DIRS('.')

# install in /usr/local/samba by default
Options.default_prefix = '/usr/local/samba'

def set_options(opt):
    opt.BUILTIN_DEFAULT('NONE')
    opt.PRIVATE_EXTENSION_DEFAULT('samba4')
    opt.RECURSE('lib/replace')
    opt.RECURSE('dynconfig')
    opt.RECURSE('lib/ldb')
    opt.RECURSE('selftest')
    opt.RECURSE('source4/lib/tls')
    opt.RECURSE('lib/nss_wrapper')
    opt.RECURSE('lib/socket_wrapper')
    opt.RECURSE('lib/uid_wrapper')
    opt.RECURSE('pidl')
    opt.RECURSE('source3')

    gr = opt.option_group('developer options')
    gr.add_option('--enable-build-farm',
                   help='enable special build farm options',
                   action='store_true', dest='BUILD_FARM')

    opt.tool_options('python') # options for disabling pyc or pyo compilation
    # enable options related to building python extensions


  [...more python...]


def main():
    from samba_wildcard import wildcard_main
    wildcard_main(wildcard_cmd)


There is no python function test() in wscript corresponding to target test. Similarly there is no function build() as required by 'make all'. Now the things start getting complicated. Before waf tries to find any of the functions corresponding to the targets, it executes set_options() function.

(For the curious ones - The code is in buildtools/.waf-xxxxx/wafadmin/Scripting.py, function prepare_impl)

Samba has extended waf functionality to make it easier for developers to specify targets/dependencies and to add special magic to detect duplicate symbol linking. One of the extension is RECURSE() function (from buildtools/wafsamba/samba_utils.py) which looks inside the specified directory for wscript files and parses them. And the line of interest is opt.RECURSE('selftest'). This tells waf to parse selftest/wscript file, which has following code.

def set_options(opt):

    opt.add_option('--enable-selftest',
                   help=("enable options necessary for selftest (default=no)"),
                   action="store_true", dest='enable_selftest', default=False)
    opt.add_option('--with-selftest-prefix',
                   help=("specify location of selftest directory (default=./st)"),
                   action="store", dest='SELFTEST_PREFIX', default='./st')

    opt.ADD_COMMAND('test', cmd_test)
    opt.ADD_COMMAND('testonly', cmd_testonly)


  [...more python...]


# main test entry point
def cmd_test(opt):
    '''Run the test suite (see test options below)'''

    # if running all tests, then force a symbol check
    env = LOAD_ENVIRONMENT()
    CHECK_MAKEFLAGS(env)
    if not Options.options.TESTS:
        Options.options.DUP_SYMBOLCHECK = True

    Scripting.commands.append('build')
    Scripting.commands.append('testonly')

Digging Deeper