Difference between revisions of "Writing Tests"

(pull writing and running Samba tests into new page)
 
m (Writing and running tests in Samba)
 
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
=Writing and running tests in Samba=
 
=Writing and running tests in Samba=
 +
 +
Samba's tests end up being run in [[The_Samba_Selftest_System|Samba's selftest system]] and [[Autobuild]].
  
 
== Background ==
 
== Background ==
Line 28: Line 30:
  
 
* [[Writing_Torture_Tests|smbtorture]] is the traditional C-based testing framework for Samba, created with the dual purpose of testing the protocol against Windows and ensuring Samba has the same behavior. <code>smbtorture</code> tests SMB and SMB2 in particular.
 
* [[Writing_Torture_Tests|smbtorture]] is the traditional C-based testing framework for Samba, created with the dual purpose of testing the protocol against Windows and ensuring Samba has the same behavior. <code>smbtorture</code> tests SMB and SMB2 in particular.
* [[Writing_Python_Tests|Python]] test scripts are used to implement integration tests in a similar way to <code>smbtorture</code>. The Python bindings that PIDL autogenerates for RPC, combined with additional LDB Python bindings for LDAP access, make it easy to write Python client code that exercises the server-side Active Directory behaviour.
+
* [[Writing_Python_Tests|Python]] test scripts are used to implement integration tests in a similar way to <code>smbtorture</code>. If the protocol under test is [[DCERPC]], then [[PIDL]] will have already auto-generated [[Python]] bindings.  Likewise LDAP is easily accessed via [[LDB]], making it easy to write Python client code that exercises the server-side Active Directory behaviour.
 
* <code>shell</code> tests can be a simple way to test the behaviour of Samba command-line tools. However, generally Python tests are preferred over <code>shell</code>, for better maintainability.
 
* <code>shell</code> tests can be a simple way to test the behaviour of Samba command-line tools. However, generally Python tests are preferred over <code>shell</code>, for better maintainability.
 
 
  
 
== Unit tests ==
 
== Unit tests ==
Line 37: Line 37:
 
Samba unit tests are generally written either in:
 
Samba unit tests are generally written either in:
  
* Python (as typical python unit tests), usually used to test Samba python libraries.
+
* [[Writing Python Tests|Python]] (as typical python unit tests), usually used to test Samba python libraries and bindings.
* The [https://cmocka.org/ cmocka] framework.
+
* The [[Writing cmocka Tests|cmocka]] framework from [https://cmocka.org/ cmocka.org].
* Part of <code>smbtorture</code> (or <code>smbtorture3</code>) as a ‘local’ test (where a server is specified but is not actually contacted).
+
* Part of <code>[[Writing Torture Tests|smbtorture]]</code> (or <code>smbtorture3</code>) as a ‘local’ test (where a server is specified but is not actually contacted).
  
 
Note that most Samba library code does not have a specific unit test, and is instead tested as part of the overall testing of Samba’s protocol implementation. There is no specific boundary to indicate where unit testing (compared with integration testing) is required, but unit testing is strongly encouraged for new and modified library code.
 
Note that most Samba library code does not have a specific unit test, and is instead tested as part of the overall testing of Samba’s protocol implementation. There is no specific boundary to indicate where unit testing (compared with integration testing) is required, but unit testing is strongly encouraged for new and modified library code.

Latest revision as of 05:14, 4 June 2020

Writing and running tests in Samba

Samba's tests end up being run in Samba's selftest system and Autobuild.

Background

Samba has been using automated tests for some time, but due to the age of the Samba project, the testing infrastructure has sometimes developed in ways that the rest of the industry did not follow. Combined with the unusual testing requirements of Samba, this means the Samba testing framework has an idiosyncratic feature set and uses approaches that are not shared by many other projects. Samba developers make efforts to adopt the standard practice, such as using Python’s unittest module in preference to a pre-existing bespoke Samba implementation. However, a significant amount of Samba-specific testing infrastructure remains.

To connect the variety of test systems into a common reporting format Samba uses the Subunit testing protocol (version 1). This protocol is both human- and machine-readable and allows us to create knownfail files listing expected test-case failures.

Note that the sub-projects of TDB, CTDB, talloc, tevent and LDB have some of their own tests declared in their own build systems. These sub-projects use a much less advanced test framework compared to Samba.


Test environments

Samba’s integration testing heavily relies on the automatic creation of a Samba network. This specialized test environment is generally referred to as a Samba ‘testenv’.

A testenv involves starting the Samba server listening on a fake network, which is established using the socket_wrapper library from cwrap. All testing is also done as a non-root user using the uid_wrapper library, also from cwrap. This allows testing without installation on a developer workstation.

Samba’s test framework uses many different types of testenv. Each testenv is customized to test a particular Samba feature or configuration.


Integration testing

A significant number of Samba tests are integration tests. This is due to the comparative-testing nature of Samba development. Integration-style tests allow developers to probe a Windows server to examine its behaviour. The Windows behaviour is then locked into an automated test that gets run against Samba as part of make test.

There are several different ways integration tests can be written:

  • smbtorture is the traditional C-based testing framework for Samba, created with the dual purpose of testing the protocol against Windows and ensuring Samba has the same behavior. smbtorture tests SMB and SMB2 in particular.
  • Python test scripts are used to implement integration tests in a similar way to smbtorture. If the protocol under test is DCERPC, then PIDL will have already auto-generated Python bindings. Likewise LDAP is easily accessed via LDB, making it easy to write Python client code that exercises the server-side Active Directory behaviour.
  • shell tests can be a simple way to test the behaviour of Samba command-line tools. However, generally Python tests are preferred over shell, for better maintainability.

Unit tests

Samba unit tests are generally written either in:

  • Python (as typical python unit tests), usually used to test Samba python libraries and bindings.
  • The cmocka framework from cmocka.org.
  • Part of smbtorture (or smbtorture3) as a ‘local’ test (where a server is specified but is not actually contacted).

Note that most Samba library code does not have a specific unit test, and is instead tested as part of the overall testing of Samba’s protocol implementation. There is no specific boundary to indicate where unit testing (compared with integration testing) is required, but unit testing is strongly encouraged for new and modified library code.