Writing Tests

Revision as of 22:05, 3 June 2020 by Abartlet (talk | contribs) (pull writing and running Samba tests into new page)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Writing and running tests in Samba

Background

Samba has been using automated tests for some time, but due to the age of the Samba project, the testing infrastructure has sometimes developed in ways that the rest of the industry did not follow. Combined with the unusual testing requirements of Samba, this means the Samba testing framework has an idiosyncratic feature set and uses approaches that are not shared by many other projects. Samba developers make efforts to adopt the standard practice, such as using Python’s unittest module in preference to a pre-existing bespoke Samba implementation. However, a significant amount of Samba-specific testing infrastructure remains.

To connect the variety of test systems into a common reporting format Samba uses the Subunit testing protocol (version 1). This protocol is both human- and machine-readable and allows us to create knownfail files listing expected test-case failures.

Note that the sub-projects of TDB, CTDB, talloc, tevent and LDB have some of their own tests declared in their own build systems. These sub-projects use a much less advanced test framework compared to Samba.


Test environments

Samba’s integration testing heavily relies on the automatic creation of a Samba network. This specialized test environment is generally referred to as a Samba ‘testenv’.

A testenv involves starting the Samba server listening on a fake network, which is established using the socket_wrapper library from cwrap. All testing is also done as a non-root user using the uid_wrapper library, also from cwrap. This allows testing without installation on a developer workstation.

Samba’s test framework uses many different types of testenv. Each testenv is customized to test a particular Samba feature or configuration.


Integration testing

A significant number of Samba tests are integration tests. This is due to the comparative-testing nature of Samba development. Integration-style tests allow developers to probe a Windows server to examine its behaviour. The Windows behaviour is then locked into an automated test that gets run against Samba as part of make test.

There are several different ways integration tests can be written:

  • smbtorture is the traditional C-based testing framework for Samba, created with the dual purpose of testing the protocol against Windows and ensuring Samba has the same behavior. smbtorture tests SMB and SMB2 in particular.
  • Python test scripts are used to implement integration tests in a similar way to smbtorture. The Python bindings that PIDL autogenerates for RPC, combined with additional LDB Python bindings for LDAP access, make it easy to write Python client code that exercises the server-side Active Directory behaviour.
  • shell tests can be a simple way to test the behaviour of Samba command-line tools. However, generally Python tests are preferred over shell, for better maintainability.


Unit tests

Samba unit tests are generally written either in:

  • Python (as typical python unit tests), usually used to test Samba python libraries.
  • The cmocka framework.
  • Part of smbtorture (or smbtorture3) as a ‘local’ test (where a server is specified but is not actually contacted).

Note that most Samba library code does not have a specific unit test, and is instead tested as part of the overall testing of Samba’s protocol implementation. There is no specific boundary to indicate where unit testing (compared with integration testing) is required, but unit testing is strongly encouraged for new and modified library code.