Writing Torture Tests: Difference between revisions

From SambaWiki
Line 123: Line 123:

* Running a samba selftest with valgrind (SMBD_VALGRIND, NMBD_VALGRIND, WINBINDD_VALGRIND)
* Running smbtorture in selftest with valgrind
make test TESTS="spoolss" VALGRIND="1" VALGRINDLOG="$(pwd)/smbtorture.%p.log"

* Running samba daemons in selftest with valgrind (SAMBA_VALGRIND, SMBD_VALGRIND, NMBD_VALGRIND, WINBINDD_VALGRIND)
make test SMBD_VALGRIND="valgrind --leak-check=full" TESTS=spoolss
make test TESTS="spoolss" SMBD_VALGRIND="valgrind --leak-check=full"

Revision as of 16:05, 29 June 2013

The smbtorture framework provided in Samba is an extremely powerful tool for exercising obscure parts of Windows protocols and determining how different server implementations, Windows among them, respond. Writing torture tests is an invaluable way to determine a server's behavior in odd scenarios. Before the release of the Microsoft protocol documentation smbtorture was the only way to determine how a Windows server would behave to packets that were legal but would not be sent by the existing Windows client.

This page attempts to layout some guidelines for development of new torture tests. Primarily this page focuses on adding tests to the source4/torture collection.

Source Layout

The source4/torture directory contains the following sub-directories:

  • basic - this directory primarily contains the original SMB tests that were ported from the source/torture test directory. These tests use the smbcli_*() wrapper functions which provide an abstracted API to the SMB protocol behavior. These tests primarily deal with older aspects of the SMBv1 protocol and should be considered legacy. If possible, new tests should not be added to this directory.
  • raw - this directory contains SMBv1 tests that exercise most aspects of the file sharing protocol by tweaking the raw bits sent in every packet. Most new SMBv1 correctness tests are added to this directory.
  • smb2 - this directory contains correctness tests for the SMBv2 file sharing protocol. Like the raw directory these tests specify each individual bits in each packet. New SMBv2 tests should be added to this directory.
  • rpc - this directory contains tests for various DCE/RPC services that Samba implements.

Best Practices

When adding new tests to smbtorture, it's easiest to copy-n-paste an existing test and modify it to behave differently. However, there are several best practices that should be followed for all newly written tests, even if they go against the existing code structure of that file.

Printing Comments

The printf() function should not be used. Instead the torture_comment() function can be used for printing messages in a torture test. Different front ends to torture, including the Samba build farm, may want to format or ignore text coming from the torture tests. Using the torture_comment function allows for this.

Similarly torture_warning() can be used to display warning messages.

Checking Failure

Older tests simply returned false to indicate that a test failed. Torture now keeps a torture_result state for each test which should be set to one of

enum torture_result {

The easiest way to set this state is by using the torture_assert_*() macros provided in lib/torture/torture.h.

These assert macros will set the torture_result state and return from the function. Often a test wants to do some cleanup on failure, such as delete test files. In these cases there are special torture_assert_*_goto() macros which will jump to a label instead of returning from a function. Some tests also have their own CHECK_*()' macros which wrap the torture_result() function. Either of these methods is acceptable.

Use Test Suites

All new tests should be added under a sub-level test suite. For instance when adding a new locking test to the raw directory it should be added in the lock.c file with an entry in the torture_raw_lock() initialization function. This allows users of smbtorture to run individual tests such as RAW-LOCK-ASYNC. New tests should not be added to the upper level suites such as RAW or BASE.

Specifying Target Behavior

Not all servers support every aspect of all Windows protocols. It's important for the test suite to be flexible enough to still offer the maximum amount of test coverage while still allowing for variation in behavior of the target server. For cases where a target server does not support a documented feature a configuration parameter can be specified which changes the error checking of a test based off that feature support.

For example not all servers support System Access Control Lists. The parametric option torture:sacl_support has been defined in the source4/torture/smbtorture.h file. Because parametric configuration is used by comparing string names, the definition of this parameter is only a comment.

/* torture:sacl_support
 * This parameter specifies whether the server supports the setting and
 * retrieval of System Access Control Lists.  This includes whether the server
 * supports the use of the SEC_FLAG_SYSTEM_SECURITY bit in the open access
 * mask.*/

Target servers which do not support this parameter then define this parameter to false in source4/torture/smbtorture.c:

        } else if (strcmp(target, "onefs") == 0) {
                lp_set_cmdline(cmdline_lp_ctx, "torture:sacl_support", "false");

Then in tests where the outcome is different based on the servers support for this feature the parameter is checked. Such as in source4/torture/basic/denytest.c:

        /* Skip all SACL related tests. */
        if ((!torture_setting_bool(tctx, "sacl_support", true)) &&
            ((cxd->cxd_access1 & SEC_FLAG_SYSTEM_SECURITY) ||
             (cxd->cxd_access2 & SEC_FLAG_SYSTEM_SECURITY)))
                goto done;

Test Verification

Running tests

The easiest way to run tests is to simply call 'make test' from either the source3 or source4 directory. This however will run all testsuites against Samba 3 or 4. The test cycle may take a long time and often it is hard to track if a specific test that has been added was run successfully. Very often you probably just want to run a single test.

  • Running a single or multiple tests in Samba 3 (make selftest)

This example would run the RPC-SVCCTL torture test from the s4 smbtorture binary against Samba 3

make test TESTS=samba3.posix_s3.rpc.svcctl

The TESTS variable also takes wildcards and just simple patterns:

make test TESTS=svcctl

results in running that very same RPC-SVCCTL test.

Many tests have subtests, if you want to run e.g. all tests for the samr DCE/RPC interface you can just call

make test TESTS=samr

which will then start the RPC-SAMR, RPC-SAMR-USERS, RPC-SAMR-PASSWORDS, etc. tests.

All tests that you can run this way need to be defined in one of the opt-in lists, all DCERPC tests using smbtorture are defined in: 'source3/script/tests/test_posix_s3.sh'

It is also possible to include specific options for the server-side configuration of samba:

make test TESTS=samr INCLUDE_CUSTOM_CONF="lanman auth = no"
  • Running a samba selftest with smbd options (SMBD_OPTIONS, NMBD_OPTIONS, WINBINDD_OPTIONS)
make test SMBD_OPTIONS="-d10" TESTS=spoolss
  • Running smbtorture in selftest with valgrind
make test TESTS="spoolss" VALGRIND="1" VALGRINDLOG="$(pwd)/smbtorture.%p.log"

  • Running samba daemons in selftest with valgrind (SAMBA_VALGRIND, SMBD_VALGRIND, NMBD_VALGRIND, WINBINDD_VALGRIND)
make test TESTS="spoolss" SMBD_VALGRIND="valgrind --leak-check=full"
  • Running a samba selftest and showing the output of the test.
make test SMBD_OPTIONS="-d2" TESTS=spoolss SUBUNIT_FORMATTER="cat"
make test SMBD_OPTIONS="-d2" TESTS=spoolss SUBUNIT_FORMATTER="tee selftest.log"
  • Running a samba test with testenv
make -j testenv SELFTEST_TESTENV=dc

Available testenv: dc, s3dc, s3member

  • Running samba best but start each server in a screen session

  • Within testenv, you can run a test and create a pcap file automatically:
export SOCKET_WRAPPER_PCAP_FILE=debug.pcap

Please note, that for each test, data will be appended to that file.

  • Disable syncs to run tests faster you can use:
TDB_NO_FSYNC=1 make test TESTS=spoolss
  • Running a single or multiple tests in Samba 4
make test TESTS=spoolss 

Test Environment

Samba 3

There are currently two test environments, a simple one to run client commands and one which attaches gdb to smbd and winbind.

  • Running the test environment
make testenv
  • Running the test enviornment with gdb attached to smbd and winbind.
make gdbtestenv SMBD_OPTIONS="-d2" WINBIND_OPTIONS="-d2"

Samba 4


Run smbtorture manually

  • Running smbtorture winreg test against a windows machine
bin/smbtorture4 ncacn_np: -UAdministrator%secret RPC-WINREG

Latest Windows

All new tests should first be run against the latest two versions of a Windows server. At the time of this writing that is Windows 2008 Server R2 / Windows 7 and Windows 2008 Server / Windows Vista. Tests should pass against both of these servers before being committed.

The behavior of the Windows server should at first be considered the correct implementation. Windows servers do have bugs and often do the wrong thing, thus some parameterization may be necessary to check for correctness based off whether the server is Windows or another target. This is covered in #Specifying_Target_Behavior.

make test failures

smbtorture is used by both the Samba3 and Samba4 'make test' self test framework. These tests are run continuously on the Samba Build Farm to prevent new code from causing regressions. Thus, it is important when writing new tests to check the behavior of the existing Samba3 and Samba4 servers before committing the test to git. This testing can be done most easily by running the 'make test' scripts for both Samba3 and Samba4 before committing a new torture test.

If failures occur against either of these servers bugs should be filed in the http://bugs.samba.org database to alert the server developers to look into these new issues.

The Samba3 'make test' framework is currently opt-in, meaning newly added torture tests will not automatically be run. However, the Samba4 'make test' framework is opt-out. Thus, if a newly added tests fails against Samba4 it will immediately cause failures to occur on the Samba Build Farm's regression tests. Thus, failing tests, after bugs have been opened for them should be added to the source4/selftest/knownfail configuration file. This let's 'make test' know that the failures are expected and should not raise any alarms on the Build Farm.