Waf: Difference between revisions

From SambaWiki
No edit summary
(93 intermediate revisions by 7 users not shown)
Line 1: Line 1:
= Experiments with using waf to build Samba =
= Using waf to build Samba =


As of March 2010, the waf build of Samba4 is now complete. We also
As I mentioned previously, I've been experimenting with using waf for
have waf build rules for libreplace, talloc, tdb, tevent, ldb and
Samba builds. So far it looks very promising. The waf maintainer is
Samba4. Kai is working on the waf build for Samba3.
being extremely helpful.


Main features:
I've added 'wscript' waf rules for libreplace, tevent, tdb, talloc and
ldb. Configure, build, install all work. The build in source4 partly works.


* rpath is used so the binaries can be run directly
I've set it up so builds appear in bin/ (so if building in
* building and configuring is fast
source4/lib/ldb then you get source4/lib/ldb/bin/ldbadd for
* full dependencies are checked, so much less need for make clean
example). This keeps the current workflow we use in Samba development.
* wrappers are provided for ./configure and make
* builds are much smaller (around 110M versus 1.1G for the old Samba4 build)
* project rules are checked to ensure no object duplication
* supports creation of working tarballs for our standalone libraries
* builds on more build-farm hosts than the existing build system
* much cleaner build rules (in wscript files). Should be easy to learn


== For Autoconf users ==
rpath is used so the binaries can be run directly (so no stuffing
about with LD_LIBRARY_PATH etc for gdb, valgrind and other testing)


In waf, Makefile.in files have been replaced with wscript_build files. Edit these to add new files/dependencies instead of the Makefile.in files that autoconf uses.
all built files (object, libs, binaries) are in bin. The source tree
is never modified. If you do "rm -rf bin" then you're back to a
pristine tree.


== Trying it out ==
I've also written a script to auto-convert our existing config.mk
files for Samba4 to wscript_build scripts. This would allow us to run
the two build systems in parallel with very little pain, as you run
one script to re-generate all the waf build scripts.


There are two ways to use the waf build. You can either use the
The full Samba4 build is not working yet. I still have quite a bit to
configure wrappers, or you can call waf directly.
do on the rules for pidl, asn1, errtable etc, plus rules for how we
will handle library dependencies.


=== Using the configure wrappers ===
waf is certainly fast. It uses full content hashing for dependencies,
but so far that hasn't been a problem. It also has a 'wafcache' which
is a ccache like system, but that applies to all builds, so it will
safely cache pidl, asn1, libraries, configure checks etc.


To build Samba using the configure wrappers do this:
The main thing I've learned from trying this is that the build rules
for Samba4 are very complex, and do not lend themselves to a simple
representation. We have 485 build targets in the Samba4 tree
(ie. subsystems, modules, binaries, libraries etc).


./configure
== Getting the code ===
make
make test


You can also use ./configure.developer as usual.
You need to do a git checkout of the waf-wip branch like this:


If you use the configure wrappers then note that the makefile is setup
git clone git://git.samba.org/tridge/samba.git samba-waf
with "JOBS=1" by default. To take advantage of parallel builds, use
cd samba-waf
something like:
git checkout waf-wip


make JOBS=4
To update your checkout when something has changed, run this:


=== Using waf directly ===
cd samba-waf
git pull --rebase


You will probably find it useful to be able to call waf directly, as
== Trying it out ==
this offers additional features. To do that follow these steps:


* setup your PATH to point at the buildtools/bin directory
This is highly experimental at the moment, but if you do want to give
it a go then try this:


export PATH=$HOME/samba-waf/buildtools/bin:$PATH
* setup your PATH to point at the buildtools/ directory in the root of the Samba sources

export PATH=$HOME/samba-waf/buildtools:$PATH


* don't install waf on your system, instead use the one in buildtools/. This ensures we are all always using the same version.
* don't install waf on your system, instead use the one in buildtools/. This ensures we are all always using the same version.

* to enable the wafcache (which speeds things up a lot), do this:

export WAFCACHE=$HOME/.wafcache
mkdir -p $WAFCACHE


* now try one of the trees that have been converted to waf. For example, to try the tdb tree do this:
* now try one of the trees that have been converted to waf. For example, to try the tdb tree do this:
Line 72: Line 58:
waf configure --enable-developer --prefix=$HOME/testprefix
waf configure --enable-developer --prefix=$HOME/testprefix
waf
waf

* If you are installing to a place that is not in your system library path, then you might like to use --enable-rpath

waf configure --enable-developer --prefix=$HOME/testprefix --enable-rpath


* that will configure tdb and build the binaries in lib/tdb/bin/. To install them use:
* that will configure tdb and build the binaries in lib/tdb/bin/. To install them use:
Line 81: Line 63:
waf install
waf install


* The other trees that have been converted are lib/replace, lib/talloc, lib/tevent and source4/lib/ldb
* The other trees that have been converted are lib/replace, lib/talloc, lib/tevent and source4/lib/ldb and source4

== Building Samba4 ==

The current waf-wip branch can partly build Samba4. This is still very experimental, but if you want
to try it, then try this:

cd samba-waf/source4
waf configure --enable-developer
waf -v # this will fail
waf -v --target=samba # this should work

The double waf command is needed because of bugs in the current wafsamba rules. This won't be
needed once the full Samba4 build is working.


== Other commands ==
== Other commands ==
Line 100: Line 69:
Try "waf --help" for help on using waf. Try "waf configure --help" for
Try "waf --help" for help on using waf. Try "waf configure --help" for
the equivalent of "./configure --help". I haven't added all of our
the equivalent of "./configure --help". I haven't added all of our
Samba configuration options yet, but I've put in a few (such as
Samba configuration options yet, but I've put in the most commonly
--enable-developer). See lib/replace/wscript for how these are added.
used ones (such as --enable-developer).


You may also find "waf -v" useful for seeing the build commands. Or
You may also find "waf -v" useful for seeing the build commands. Or
Line 111: Line 80:
waf configure --enable-developer -b devbuild
waf configure --enable-developer -b devbuild


It ends up creating the binaries in devbuild instead of the default of bin/.
While will set you up for creating the binaries in devbuild instead of
the default of bin/.


If you want to debug why something isn't working, then you can either increase the verbosity of waf with something like "waf -vvv", or you can enable tracing of a particular phase of the build. For example, you could trace the build dependencies like this:
If you want to debug why something isn't working, then you can either
increase the verbosity of waf with something like "waf -vvv", or you
can enable tracing of a particular phase of the build. For example,
you could trace the build dependencies like this:


waf -v --zone=deps
waf -v --zone=deps


== Developing with waf ==
It is sometimes useful to disable the wafcache when debugging. use the --nocache option for that:


With the waf based build we get a number of new features that we
waf -v --nocache clean build
didn't have in the old build system.

=== creating distribution tarballs: waf dist ===

The waf build automates the process of creating distribution
tarballs. It also automatically bundles the needed build tools and the
library dependencies. For example:

cd lib/tevent
waf dist

Will create tevent-0.9.8.tar.gz. If you unpack that tarball then
you'll find it has automatically included the talloc and libreplace
libraries in the distribution. It has also setup a autogen-waf.sh
which will generate a working configure script and makefile.

The command 'waf distcheck' is also useful. It will create a distribution
tarball, then unpack it in a temporary directory, and configure, build and install
it (in another temporary directory). It will then run 'waf uninstall' and check
that no install files remain.

=== working with libraries ===

One of the main challenges we had with the previous build system was
how we built and used shared libraries. We had difficulty getting a
system in place that worked well both for the package maintainers in
distributions, and for people building Samba themselves.

To address these problems the waf build uses a number of strategies:

* rpath is used to make the lives of developers easier
* 'library bundling' is used to ensure libraries don't conflict with system libraries
* configure options are provided to customize the library handling

Perhaps the easiest way to explain the new system is by explaining
each of the key configure options, and give common use cases.

==== configure --bundled-libraries=LIST ====

The --bundled-libraries option allows you to choose which libraries
are 'bundled' with the build. A bundled library is a in-tree library
which may have a system equivalent on some systems. The waf build
process needs to decide for each library whether it will bundle it or
not.

If the configure process does decide to bundle a library, then it will
automatically append a suffix onto the library name. The default
suffix is the name of the package you are building. For example, if
when building Samba4 you bundle the tdb library, then it will build a
libtdb-samba4.so shared library.

The reason for the renaming is that it allows the libraries to be
installed safely. If we left the name as libtdb.so, then there would
be no way (without rpath) to run the newly built tdb library for
Samba, but leave the rest of the system using the system supplied tdb
library.

The default is to try to use system libraries if they are at least the
same version as the in-tree version. This means that if we increase
the version number of our in-tree copy of tdb, that we will prefer
that in-tree copy as a bundled library until the system version
catches up.

The following examples show how to control this process:

waf configure --bundled-libraries=NONE

The 'NONE' option is what many distribution packagers will choose. It
tells waf not to bundle any libraries. That means that if the
configure process can't find a suitable system library (of the right
version) the configure will fail.

waf configure --bundled-libraries=ALL

The 'ALL' option means to bundle all libraries. That means any
equivalent system libraries for our in-tree libraries are ignored.

waf configure --bundled-libraries=ldb,NONE

That means to bundle ldb, but to get all other libraries from the
system.

waf configure --bundled-libraries=!tdb,ALL

That means to bundle all libraries except tdb. In that case tdb comes
from the system.

The bundled library checks also include a library dependency check. It
is considered an error to get a library from the system, but to get a
dependency of that library as a bundled library. To give an example of
why that matters, consider this configure command:

waf configure --bundled-libraries=!tevent,ALL

That would tell configure to bundle all libraries except tevent. The
problem with this is that tevent depends on talloc. So if we did used
the above settings, then we would end up linking in talloc twice, once
from the system libtevent.so, and once from the bundled
libtalloc-samba4.so.

The configure process detects this error, and gives an error.

==== configure --builtin-libraries=LIST ====

The --builtin-libraries configure option allows you to control what
libraries are directly (statically) linked into the program, instead
of being linked as shared libraries.

You might, for example, want to link libreplace directly into Samba4,
in which case you would use:

waf configure --builtin-libraries=replace

For Samba4 the default is to build all libraries as shared
libraries. The default for standalone builds of tdb, talloc, tevent
and ldb are to build libreplace as a builtin, with the other
dependent libraries built as shared libraries.

All of this is controlled in the wscript files with statements like
this:

if conf.CHECK_BUNDLED_SYSTEM('tevent', minversion=VERSION,
onlyif='talloc', implied_deps='replace talloc'):
conf.define('USING_SYSTEM_TEVENT', 1)

that says that when building a standalone tevent, we should only use
the system version of tevent if we also have the system version of
talloc.

==== configure --bundled-library-extension ====

The --bundled-library-extension option allows you to control the extra
extension added to the names of bundled libraries. For example, if you
wanted bundled libraries to be called (for example) libldb-s4.so, then
use:

waf configure --bundled-library-extension=s4

You can also provide a list of libraries not to apply the extension
to. For example:

waf configure --bundled-extension-exception=talloc

would tell configure not to add a bundling extension when building
talloc.

==== configure --minimum-library-version ====

By default the waf build will only use an installed system library if
it is at least the same version as the in-tree library. This maximizes
the chance that the library that is linked to will have been properly
tested by the developers.

You can override this choice by using --minimum-library-version, for
example:

waf configure --mininum-library-version=talloc:2.0.1,tdb:1.1.5

that would change the minimum library version for talloc to 2.0.1, and
for tdb to 1.1.5

It is recommended that you don't use this option without a very good
reason, as if you use it you risk creating a build that doesn't
incorporate necessary bugfixes from recent changes in the libraries.

=== controlling rpath ===

The 'rpath' facility of shared libraries and binaries is a way to
control where the binary/library goes to look for its dependencies. It
is like building a LD_LIBRARY_PATH directly into the binary/library.

rpath is particularly useful for developers, as it allows the running
of build binaries without mucking about with LD_LIBRARY_PATH.

The default for the waf build is to use rpath both for the build and
the install. This is the most useful option for people who build Samba
themselves.

For distribution packagers, rpath is rarely wanted, and is often
forbidden. So if you are making a distribution of Samba or one of our
standalone libraries, you should use:

waf configure --disable-rpath

You can also disable it just for install, if that is what you prefer:

waf configure --disable-rpath-install

=== Out of tree builds ===

waf supports out of tree builds using the -b option to configure. For
example:

cd lib/tdb
waf configure -b /tmp/tdbbuild
waf

That puts all object files, generated files and binaries in
/tmp/tdbbuild.

This works nicely for simple packages like tdb, but it gets trickier
for full Samba builds. The current problem is with our testsuite,
which has some hard-coded places where it assumes binaries in
bin/. We also have a problem with the gen_ndr directory, which breaks
the out of tree build of IDL files.

In the future we should be able to support out of tree builds for
Samba4, but not just yet.

=== cross-compiling ===

You can build for non-host architectures by using the
cross-compilation features of the build.

There are two approaches to cross compiling with the waf build:

* using a --cross-execute emulator
* using a --cross-answers file

In either case you will need a cross-compiler setup installed.

==== Using --cross-execute ====

If you have an emulator setup to run cross-compiled binaries then you
can use --cross-execute. This will allow you to accurately configure
your build with the correct answers.

For example, on a Debian derived system for an ARM target you would
install gcc-4.3-arm-linux-gnu, linux-kernel-headers-arm-cross and
qemu-arm-static. That will give you enough to create ARM binaries.

You may find the following apt sources useful for this:

deb http://mirror.csclub.uwaterloo.ca/emdebian/ stable main
deb-src http://mirror.csclub.uwaterloo.ca/emdebian/ stable main

With those installed, you would then configure like this:

CC=arm-linux-gnu-gcc waf configure --cross-compile --cross-execute='qemu-arm-static -L /usr/arm-linux-gnu'

that tells the build to use the arm-linux-gnu-gcc compiler, and to
cross-execute the test binaries during the configure stage using the
qemu-arm-static tool. By using a --cross-execute tool you can ensure
that all your configure tests can run correctly, which avoids the
problem of having to work out things like the size of various target
architecture types.

After the configure, run waf as usual to build. You can then test the
resulting binaries using (for example):

qemu-arm-static -L /usr/arm-linux-gnu bin/ldbadd

and it will run using qemu.

==== Using --cross-answers ====

If you don't have a emulator for your target machine, you will need to
use the --cross-answers option. When you specify something like
--cross-answers=target.txt the build will place unknown configure
tests in target.txt. You then need to edit that file, and change any
UNKNOWN answers with the correct answers for your target.

For example, you could do this:

CC=arm-linux-gnu-gcc waf configure --cross-compile --cross-answers=arm.txt

That will initially create a file like this:

Checking simple C program: UNKNOWN

The configure stopped at that point as it needs to know if a simple C
program can run before it will continue. If you look in bin/config.log
(or in bin/.conf_check_0/) you will see the source code for the program
that it was testing.

To continue the configure process, you would edit arm.txt and replace
UNKNOWN with OK like this:

Checking simple C program: OK

The configure will again fail, and you will end up with this in
arm.txt:

Checking simple C program: OK
building library support: UNKNOWN
Checking for large file support: UNKNOWN
Checking for -D_FILE_OFFSET_BITS=64: UNKNOWN
Checking for WORDS_BIGENDIAN: UNKNOWN
Checking size of char: UNKNOWN
Checking size of int: UNKNOWN
Checking size of long long: UNKNOWN
Checking size of long: UNKNOWN
Checking size of off_t: UNKNOWN
Checking size of short: UNKNOWN
Checking size of size_t: UNKNOWN
Checking size of ssize_t: UNKNOWN
Checking size of dev_t: UNKNOWN
Checking size of ino_t: UNKNOWN
Checking size of time_t: UNKNOWN
Checking size of void*: UNKNOWN
Checking for C99 vsnprintf: UNKNOWN
Checking for HAVE_SECURE_MKSTEMP: UNKNOWN

Note that only the tests that involve running an executable on the target have been added. The
other tests are able to be completed by running on the host, and thus do not need a cross-answers
entry.

You then should replace all the UNKNOWN values with the correct values
for your platform. For example:

Checking simple C program: OK
building library support: OK
Checking for large file support: OK
Checking for -D_FILE_OFFSET_BITS=64: OK
Checking for WORDS_BIGENDIAN: OK
Checking size of char: "1"
Checking size of int: "4"
Checking size of long long: "8"
Checking size of long: "4"
Checking size of off_t: "8"
Checking size of short: "2"
Checking size of size_t: "4"
Checking size of ssize_t: "4"
Checking size of dev_t: "4"
Checking size of ino_t: "4"
Checking size of time_t: "4"
Checking size of void*: "4"
Checking for C99 vsnprintf: OK
Checking for HAVE_SECURE_MKSTEMP: OK

The right hand side of each line can be in one of several formats, all
of which map to a exit code and a returned string. There are also
shortcut forms:

* a OK means that the test should return a exit code of 0, and an empty string
* a FAIL or NO means that the test returns a exit code of 255, and an empty string
* a UNKNOWN means the test has an unknown result
* a "STRING" or 'STRING' means the test has an exit code of 0, and the given string

The general form is:

Some message: (RETCODE, RETSTRING)

here are some examples:

Some message: (0, "")
Some message: (23, "")
Some message: (0, "a string")
Some message: (1, "another string")

If your configure step completes, but there are still UNKNOWN values in your answers file, then the following message will be displayed:

Cross answers file arm.txt is incomplete

and you will need to continue with the iterative process until all answers are filled in

=== ABI Checking ===

The waf build has the ability to check that developers are not
changing the ABI of libraries. This is enabled by default if you use
configure with the --enable-developer option, or if you use the
--abi-check option. You can disable this ABI checking by using
--abi-check-disable

The way the ABI checking works is this:

* A library declaration that uses ABI checking gets two extra options, abi_file and abi_match.

* The ABI signatures are generated using the buildtools/scripts/abi_gen.sh, which uses nm and gdb to generate a set of ABI signatures

* when the library is linked, the build system will check if the ABI signatures have changed. If they have, then it will suggest modifying the major version number if a symbol has been changed or removed, and the minor version number if a symbol has been added. The build will fail.

* if the ABI has changed and the build fails, then you should update the library version (the VERSION variable in the library wscript) and then re-run the build. That will regenerate the new signatures file. This should be committed to the git tree.

* if the ABI has changed for an existing ABI version (eg. change a symbol) and you are really sure you wish to overwrite the old ABI signatures, then you can build with the --abi-update option. That will cause the new ABI signatures to be written out to the ABI file for the existing ABI version.

* ABI checking will only be done if your system supports the -fvisibility=hidden compiler option, as otherwise spurious symbols may appear in the ABI signatures

As an example, if you were to change the 2nd argument of talloc_is_parent() to be non-const, you would see an error message like this:

[13/22] abi_check: lib/talloc/bin/default/lib/talloc/libtalloc.so
libtalloc.so: symbol talloc_is_parent has changed - please update major version
old_signature: int (const void *, const void *)
new_signature: int (const void *, void *)
Waf: Leaving directory `/home/tridge/samba/git/combined/lib/talloc/bin'
Build failed: ABI for libtalloc.so has changed - please fix library version then build with --abi-update
See http://wiki.samba.org/index.php/Waf#ABI_Checking for more information
make: *** [all] Error 1

in that case you would need to edit VERSION in lib/talloc/wscript, then run "waf --abi-update"
to regenerate the ABI signatures.

If you added a new symbol talloc_is_parent2() then you would see this:

[13/22] abi_check: lib/talloc/bin/default/lib/talloc/libtalloc.so
libtalloc.so: symbol talloc_is_parent2 has been added - please mark it _PRIVATE_ or update minor version
signature: int (const void *, const void *)
Waf: Leaving directory `/home/tridge/samba/git/combined/lib/talloc/bin'
Build failed: ABI for libtalloc.so has changed - please fix library version then build with --abi-update
See http://wiki.samba.org/index.php/Waf#ABI_Checking for more information
make: *** [all] Error 1

in that case you would need to either mark talloc_is_parent2() as
being _PRIVATE_, or you could edit VERSION in lib/talloc/wscript, then
rebuild to regenerate the ABI signatures.

Here is an example library declaration with ABI checking for talloc:

bld.SAMBA_LIBRARY('talloc',
'talloc.c',
deps='replace',
abi_file='ABI/talloc-%s.sigs' % VERSION,
abi_match='talloc* _talloc*',
hide_symbols=True,
vnum=VERSION)

this will place the ABI file in ABI/talloc-2.0.2.sigs for the 2.0.2
version of the library. A separate signatures file is generated for
each library version. The abi_match option means that only symbols
starting with talloc or _talloc will be considered as part of the ABI.

The hide_symbols=True option tells the build to enable the
-fvisibility=hidden compiler option (if supported) for building this
library. That is used in combination with the _PUBLIC_ and _PRIVATE_
attributes on functions to ensure that symbols are not exposed in
libraries when they shouldn't be. The eventual aim will be to enable
symbol hiding in all of Samba, but we first need to ensure that all
ABI symbols in libraries are marked as _PUBLIC_.

In some cases you want to enable -fvisibility=hidden only when a
library is a builtin library. For example, we want to hide symbols
from libreplace when it is builtin to another library, but expose them
if libreplace is built as a standalone library. To do that use this:

bld.SAMBA_LIBRARY('replace',
source=REPLACE_SOURCE,
group='base_libraries',
hide_symbols=bld.BUILTIN_LIBRARY('replace'),
deps='crypt dl nsl socket')

== Packaging with waf ==

If you are the distribution maintainer of one of the packages dervived
from the Samba git tree, then you will probably want to use the
following configure options:

waf configure --disable-rpath --bundled-libraries=NONE

The above options will disable use of rpath (for both build and
install), and will force the use of all system libraries. That means
you will first need to ensure that your system has up to date versions
of all the dependent libraries, otherwise the configure will fail.

You may also want to control the number of builtin libraries. See the
section on builtin libraries above.

Apart from that, packaging with a waf build is very similar to
packaging with a autotools based system.

You may also want to add JOBS=4 or similar to the make command to get
build parallelism. The makefile based waf build defaults to one CPU.


== How it works ==
== How it works ==
Line 126: Line 560:
file. That file can depend on other project wscript files. I've kept
file. That file can depend on other project wscript files. I've kept
the wscript files for each project simple by using a 'waf tool' called
the wscript files for each project simple by using a 'waf tool' called
'wafsamba.py', which is an add-on for waf that sets up the waf
'wafsamba', which is in buildtools/wafsamba. Wafsamba is an add-on for
configuration and build commands to follow the same sorts of
waf that sets up the waf configuration and build commands to follow
conventions that we have in the existing Samba4 build system.
the same sorts of conventions that we have in the existing Samba4
build system.


If you want to have a look at how this was done, then look in
If you want to have a look at how this was done, then look in
Line 142: Line 577:
* the 'wafsamba' waf tool which allows us to describe samba build rules succinctly are here [http://git.samba.org/?p=tridge/samba.git;a=blob;f=buildtools/wafsamba/wafsamba.py;hb=waf-wip buildtools/wafsamba/wafsamba.py]
* the 'wafsamba' waf tool which allows us to describe samba build rules succinctly are here [http://git.samba.org/?p=tridge/samba.git;a=blob;f=buildtools/wafsamba/wafsamba.py;hb=waf-wip buildtools/wafsamba/wafsamba.py]


== Adding new commands ==
== Documentation ==


waf itself is documented in a online book
In Samba we are used to a number of non-standard make targets, such as 'make quicktest'. We can add the same targets using waf. Even better, when you add a new target in waf it shows up in 'waf --help'. To give an example, I've added a selftest/wscript here [http://git.samba.org/?p=tridge/samba.git;a=blob;f=source4/selftest/wscript;hb=waf-wip source4/selftest/wscript]. As you can see, it adds a 'test' and 'quicktest' target. Now if we run 'waf --help' we get this:
here: http://freehackers.org/~tnagy/wafbook/single.html


The build rules I've added for Samba also need to be documented. I've
Main commands (example: ./waf build -j4)
started this process using pydoctor.
clean : removes the build files
configure: configures the project
dist : makes a tarball for redistributing the sources
distcheck: checks if the sources compile (tarball from 'dist')
distclean: removes the build directory
install : installs the build files
quicktest: Run the quick test suite
test : Run the full test suite
uninstall: removes the installed files


= Parallel builds =
The commands themselves are arbitrary python. The simplest first approach will be to call our existing
perl based testsuite. Then if we wish to convert it to python later we can.


One nice thing about the use of waf is we gain parallel builds. On a
== Documentation ==
16 CPU machine a complete clean Samba4 build takes just 35s, including
rebuilding all of the IDL files. It takes over 5 minutes on the same
machine with the old build system.


Here is a graph of the build time versus -j flag
waf itself is documented in a online book

here: http://freehackers.org/~tnagy/wafbook/single.html
[[Image:16-cpus.png]]

and here is a picture which shows the degrees of parallelism at different points in the build
with 4 CPUs

[[Image:4-way-parallelism.png]]

= TODO =

A short list of work items, not complete!

== detect lack of gdb on developer build ==

Waf should error out earlier if ./configure.developer is run without gdb available for the ABI scripts.

== detect system changes ==

It would be nice to detect changing system headers/libraries and give an error at build stage

== Rules for yapp, yacc and lex ==

We need rules to regenerate files with yapp (for pidl) if the grammar has changed.

We need rules to regenerate files with yacc and lex (for heimdal),
here we need to only allow trusted combinations of bison and flex.

If the system has this tools we need to have rules for them,
otherwise we need a way to give an error if the "source" has changed
but we're not able to regenerate.

update: done for yapp

== add shared lib support for other compilers ==

The AIX, IRIX and TRU64 compilers can generate shared libs. We need
to support that and test that it works correctly.

== add better optimizations options ==

we should add more control over the use of -O2, -O1 etc in configure

== build static libraries ==

the standalone libraries need to be built as static libraries as well

= COMPLETED TODO ITEMS =

== speedup null build ==

the null build now takes about 1.6s which is good enough

== fix 'waf test' ==

all s4 tests now pass with the waf build

== check each compiler option ==

for options like --enable-developer we need to check the compiler supports
the various options.

The flags should only be used in the build and not within configure checks.

== get the waf build working in the build farm ==

This will test the portability quite well

its passing on some hosts in the build farm. The others are being
worked on

== get git revision in logs and -V output ==

bin/samba -V should show the git revision

== waf configure --help ==

waf configure --help should display 'build' as possible command

== implement waf install ==

We need to call the same install scripts we used from the old build,
at least initially.

all done now

== add --no-bundled-libraries ==

done, as --bundled-libraries=LIST

== implement LIBRARY->LIBRARY dependency reduction ==

We want object files used in common between libraries that depend on
each other to be removed from the parent library, unless the parent library
exports the symbols publicly and the child library is a private library

done. Installed size of s4 with waf is now down to about 110M

== make waf builds works from subdirectories ==

You could be able to "cd source4/client" and run waf

done

== waf dist support ==

done. We now build working tarballs for tdb, ldb, talloc, replace,
tevent and samba4

== Support waf test --gdb ==
We need to have a replacement for the old 'make gdbtest' and 'make gdbtest-env'.

done

== uuid module ==

The uuid module in scripting/python should only be built if the python version in use is <= 2.4. Newer versions of Python include this module.

done

== parse MAKEFLAGS ==

If someone uses 'make -j N' then that sets MAKEFLAGS in waf. It would be nice
to parse that and setup the right number of jobs, for people who want to do
parallel builds with 'make' instead of 'waf'

done: we parse -j and -k now

== check RPM build of the Samba libs ==

check that the spec files work with minimal changes

done: working spec files sent to Simo

== Add ABI checks to waf dist ==

When preparing releases make sure ABI/API are checked for breakage so that it
can be handled appropriately by either fixing the code or eventually increasing
the .so version for libraries marked stable.

done: see ABI Checking above

== implement the non-emulator based cross-compile method ==

See the proposal here: http://lists.samba.org/archive/samba-technical/2010-April/070563.html
and here: http://lists.samba.org/archive/samba-technical/2010-April/070564.html

done: see cross-answers section above

== detect configure changes ==

We need to detect configure related changes to wscripts and give an error at build stage.

done: added --enable-auto-reconfigure option. See discussion on list

== ldb standalone build install ldb.inst.h ==

the ldb standalone build installs a 'ldb.inst.h' file rather than a 'ldb.h' file

== ldb standalone build needs to install manual pages ==

the ldb standalone install no longer installs manual pages

== detect lack of gdb on developer build ==

Waf should error out earlier if ./configure.developer is run without gdb available for the ABI scripts.

== use of system heimdal ==

it is not yet possible to use the system-installed heimdal when using the waf build, it will always use the internal heimdal at the moment.

there is some work work in progress in heimdal_build/wscript_configure to support this.


The build rules I've added for Samba would also need to be documented
if we adopt waf, probably by annotating wafsamba.py.


== Run autogen-waf.sh within waf dist ==
== Converting Samba4 ==
Tarballs should come with ./configure already available, run autogen-waf.sh as
part of waf dist so that the only thing users need to do is ./configure && make


== support public_deps symbol hiding ==
Converting libreplace, talloc, ldb etc was easy. The hard bit comes
when trying to do a full Samba4 build. If you want to see what I've
doing for that, have a look in buildtools/mktowscript/. That contains
an extremely hackish perl script that converts our existing .mk files
to wscript_build files.


We need to use the linker scripts to hide non-public symbols in libraries
You can see an example auto-generated waf build file for the dsdb/
subsystem of Samba4 here [http://git.samba.org/?p=tridge/samba.git;a=blob;f=source4/dsdb/wscript_build;hb=waf-wip source4/dsdb/wscript_build]


== shared modules are empty ==
I think the resulting auto-generated file compares quite favourably to the original config.mk that it was generated from [http://git.samba.org/?p=tridge/samba.git;a=blob;f=source4/dsdb/config.mk;hb=waf-wip source4/dsdb/config.mk]


.so files created for modules do not contain any symbols, making them unusable.
There is a lot more work to do before we can build Samba3 and Samba4 fully with waf, but this should give you some idea of the approach I am taking.

Revision as of 14:59, 3 January 2014

Using waf to build Samba

As of March 2010, the waf build of Samba4 is now complete. We also have waf build rules for libreplace, talloc, tdb, tevent, ldb and Samba4. Kai is working on the waf build for Samba3.

Main features:

  • rpath is used so the binaries can be run directly
  • building and configuring is fast
  • full dependencies are checked, so much less need for make clean
  • wrappers are provided for ./configure and make
  • builds are much smaller (around 110M versus 1.1G for the old Samba4 build)
  • project rules are checked to ensure no object duplication
  • supports creation of working tarballs for our standalone libraries
  • builds on more build-farm hosts than the existing build system
  • much cleaner build rules (in wscript files). Should be easy to learn

For Autoconf users

In waf, Makefile.in files have been replaced with wscript_build files. Edit these to add new files/dependencies instead of the Makefile.in files that autoconf uses.

Trying it out

There are two ways to use the waf build. You can either use the configure wrappers, or you can call waf directly.

Using the configure wrappers

To build Samba using the configure wrappers do this:

./configure
make
make test

You can also use ./configure.developer as usual.

If you use the configure wrappers then note that the makefile is setup with "JOBS=1" by default. To take advantage of parallel builds, use something like:

make JOBS=4

Using waf directly

You will probably find it useful to be able to call waf directly, as this offers additional features. To do that follow these steps:

  • setup your PATH to point at the buildtools/bin directory
export PATH=$HOME/samba-waf/buildtools/bin:$PATH
  • don't install waf on your system, instead use the one in buildtools/. This ensures we are all always using the same version.
  • now try one of the trees that have been converted to waf. For example, to try the tdb tree do this:
cd lib/tdb
waf configure --enable-developer --prefix=$HOME/testprefix
waf
  • that will configure tdb and build the binaries in lib/tdb/bin/. To install them use:
waf install
  • The other trees that have been converted are lib/replace, lib/talloc, lib/tevent and source4/lib/ldb and source4

Other commands

Try "waf --help" for help on using waf. Try "waf configure --help" for the equivalent of "./configure --help". I haven't added all of our Samba configuration options yet, but I've put in the most commonly used ones (such as --enable-developer).

You may also find "waf -v" useful for seeing the build commands. Or use "waf -v clean build" to clean and build a tree, with the commands shown. Use "waf -p" to show a progress bar on the build.

This command illustrates the use of multiple build directories:

waf configure --enable-developer -b devbuild

While will set you up for creating the binaries in devbuild instead of the default of bin/.

If you want to debug why something isn't working, then you can either increase the verbosity of waf with something like "waf -vvv", or you can enable tracing of a particular phase of the build. For example, you could trace the build dependencies like this:

 waf -v --zone=deps

Developing with waf

With the waf based build we get a number of new features that we didn't have in the old build system.

creating distribution tarballs: waf dist

The waf build automates the process of creating distribution tarballs. It also automatically bundles the needed build tools and the library dependencies. For example:

cd lib/tevent
waf dist

Will create tevent-0.9.8.tar.gz. If you unpack that tarball then you'll find it has automatically included the talloc and libreplace libraries in the distribution. It has also setup a autogen-waf.sh which will generate a working configure script and makefile.

The command 'waf distcheck' is also useful. It will create a distribution tarball, then unpack it in a temporary directory, and configure, build and install it (in another temporary directory). It will then run 'waf uninstall' and check that no install files remain.

working with libraries

One of the main challenges we had with the previous build system was how we built and used shared libraries. We had difficulty getting a system in place that worked well both for the package maintainers in distributions, and for people building Samba themselves.

To address these problems the waf build uses a number of strategies:

  • rpath is used to make the lives of developers easier
  • 'library bundling' is used to ensure libraries don't conflict with system libraries
  • configure options are provided to customize the library handling

Perhaps the easiest way to explain the new system is by explaining each of the key configure options, and give common use cases.

configure --bundled-libraries=LIST

The --bundled-libraries option allows you to choose which libraries are 'bundled' with the build. A bundled library is a in-tree library which may have a system equivalent on some systems. The waf build process needs to decide for each library whether it will bundle it or not.

If the configure process does decide to bundle a library, then it will automatically append a suffix onto the library name. The default suffix is the name of the package you are building. For example, if when building Samba4 you bundle the tdb library, then it will build a libtdb-samba4.so shared library.

The reason for the renaming is that it allows the libraries to be installed safely. If we left the name as libtdb.so, then there would be no way (without rpath) to run the newly built tdb library for Samba, but leave the rest of the system using the system supplied tdb library.

The default is to try to use system libraries if they are at least the same version as the in-tree version. This means that if we increase the version number of our in-tree copy of tdb, that we will prefer that in-tree copy as a bundled library until the system version catches up.

The following examples show how to control this process:

waf configure --bundled-libraries=NONE

The 'NONE' option is what many distribution packagers will choose. It tells waf not to bundle any libraries. That means that if the configure process can't find a suitable system library (of the right version) the configure will fail.

waf configure --bundled-libraries=ALL

The 'ALL' option means to bundle all libraries. That means any equivalent system libraries for our in-tree libraries are ignored.

waf configure --bundled-libraries=ldb,NONE

That means to bundle ldb, but to get all other libraries from the system.

waf configure --bundled-libraries=!tdb,ALL

That means to bundle all libraries except tdb. In that case tdb comes from the system.

The bundled library checks also include a library dependency check. It is considered an error to get a library from the system, but to get a dependency of that library as a bundled library. To give an example of why that matters, consider this configure command:

waf configure --bundled-libraries=!tevent,ALL

That would tell configure to bundle all libraries except tevent. The problem with this is that tevent depends on talloc. So if we did used the above settings, then we would end up linking in talloc twice, once from the system libtevent.so, and once from the bundled libtalloc-samba4.so.

The configure process detects this error, and gives an error.

configure --builtin-libraries=LIST

The --builtin-libraries configure option allows you to control what libraries are directly (statically) linked into the program, instead of being linked as shared libraries.

You might, for example, want to link libreplace directly into Samba4, in which case you would use:

waf configure --builtin-libraries=replace

For Samba4 the default is to build all libraries as shared libraries. The default for standalone builds of tdb, talloc, tevent and ldb are to build libreplace as a builtin, with the other dependent libraries built as shared libraries.

All of this is controlled in the wscript files with statements like this:

   if conf.CHECK_BUNDLED_SYSTEM('tevent', minversion=VERSION,
                                onlyif='talloc', implied_deps='replace talloc'):
       conf.define('USING_SYSTEM_TEVENT', 1)

that says that when building a standalone tevent, we should only use the system version of tevent if we also have the system version of talloc.

configure --bundled-library-extension

The --bundled-library-extension option allows you to control the extra extension added to the names of bundled libraries. For example, if you wanted bundled libraries to be called (for example) libldb-s4.so, then use:

waf configure --bundled-library-extension=s4

You can also provide a list of libraries not to apply the extension to. For example:

waf configure --bundled-extension-exception=talloc

would tell configure not to add a bundling extension when building talloc.

configure --minimum-library-version

By default the waf build will only use an installed system library if it is at least the same version as the in-tree library. This maximizes the chance that the library that is linked to will have been properly tested by the developers.

You can override this choice by using --minimum-library-version, for example:

waf configure --mininum-library-version=talloc:2.0.1,tdb:1.1.5

that would change the minimum library version for talloc to 2.0.1, and for tdb to 1.1.5

It is recommended that you don't use this option without a very good reason, as if you use it you risk creating a build that doesn't incorporate necessary bugfixes from recent changes in the libraries.

controlling rpath

The 'rpath' facility of shared libraries and binaries is a way to control where the binary/library goes to look for its dependencies. It is like building a LD_LIBRARY_PATH directly into the binary/library.

rpath is particularly useful for developers, as it allows the running of build binaries without mucking about with LD_LIBRARY_PATH.

The default for the waf build is to use rpath both for the build and the install. This is the most useful option for people who build Samba themselves.

For distribution packagers, rpath is rarely wanted, and is often forbidden. So if you are making a distribution of Samba or one of our standalone libraries, you should use:

waf configure --disable-rpath

You can also disable it just for install, if that is what you prefer:

waf configure --disable-rpath-install

Out of tree builds

waf supports out of tree builds using the -b option to configure. For example:

cd lib/tdb
waf configure -b /tmp/tdbbuild
waf

That puts all object files, generated files and binaries in /tmp/tdbbuild.

This works nicely for simple packages like tdb, but it gets trickier for full Samba builds. The current problem is with our testsuite, which has some hard-coded places where it assumes binaries in bin/. We also have a problem with the gen_ndr directory, which breaks the out of tree build of IDL files.

In the future we should be able to support out of tree builds for Samba4, but not just yet.

cross-compiling

You can build for non-host architectures by using the cross-compilation features of the build.

There are two approaches to cross compiling with the waf build:

  • using a --cross-execute emulator
  • using a --cross-answers file

In either case you will need a cross-compiler setup installed.

Using --cross-execute

If you have an emulator setup to run cross-compiled binaries then you can use --cross-execute. This will allow you to accurately configure your build with the correct answers.

For example, on a Debian derived system for an ARM target you would install gcc-4.3-arm-linux-gnu, linux-kernel-headers-arm-cross and qemu-arm-static. That will give you enough to create ARM binaries.

You may find the following apt sources useful for this:

deb http://mirror.csclub.uwaterloo.ca/emdebian/ stable main
deb-src http://mirror.csclub.uwaterloo.ca/emdebian/ stable main

With those installed, you would then configure like this:

CC=arm-linux-gnu-gcc waf configure --cross-compile --cross-execute='qemu-arm-static -L /usr/arm-linux-gnu'

that tells the build to use the arm-linux-gnu-gcc compiler, and to cross-execute the test binaries during the configure stage using the qemu-arm-static tool. By using a --cross-execute tool you can ensure that all your configure tests can run correctly, which avoids the problem of having to work out things like the size of various target architecture types.

After the configure, run waf as usual to build. You can then test the resulting binaries using (for example):

qemu-arm-static -L /usr/arm-linux-gnu bin/ldbadd

and it will run using qemu.

Using --cross-answers

If you don't have a emulator for your target machine, you will need to use the --cross-answers option. When you specify something like --cross-answers=target.txt the build will place unknown configure tests in target.txt. You then need to edit that file, and change any UNKNOWN answers with the correct answers for your target.

For example, you could do this:

CC=arm-linux-gnu-gcc waf configure --cross-compile --cross-answers=arm.txt

That will initially create a file like this:

Checking simple C program: UNKNOWN

The configure stopped at that point as it needs to know if a simple C program can run before it will continue. If you look in bin/config.log (or in bin/.conf_check_0/) you will see the source code for the program that it was testing.

To continue the configure process, you would edit arm.txt and replace UNKNOWN with OK like this:

Checking simple C program: OK

The configure will again fail, and you will end up with this in arm.txt:

Checking simple C program: OK
building library support: UNKNOWN
Checking for large file support: UNKNOWN
Checking for -D_FILE_OFFSET_BITS=64: UNKNOWN
Checking for WORDS_BIGENDIAN: UNKNOWN
Checking size of char: UNKNOWN
Checking size of int: UNKNOWN
Checking size of long long: UNKNOWN
Checking size of long: UNKNOWN
Checking size of off_t: UNKNOWN
Checking size of short: UNKNOWN
Checking size of size_t: UNKNOWN
Checking size of ssize_t: UNKNOWN
Checking size of dev_t: UNKNOWN
Checking size of ino_t: UNKNOWN
Checking size of time_t: UNKNOWN
Checking size of void*: UNKNOWN
Checking for C99 vsnprintf: UNKNOWN
Checking for HAVE_SECURE_MKSTEMP: UNKNOWN

Note that only the tests that involve running an executable on the target have been added. The other tests are able to be completed by running on the host, and thus do not need a cross-answers entry.

You then should replace all the UNKNOWN values with the correct values for your platform. For example:

Checking simple C program: OK
building library support: OK
Checking for large file support: OK
Checking for -D_FILE_OFFSET_BITS=64: OK
Checking for WORDS_BIGENDIAN: OK
Checking size of char: "1"
Checking size of int: "4"
Checking size of long long: "8"
Checking size of long: "4"
Checking size of off_t: "8"
Checking size of short: "2"
Checking size of size_t: "4"
Checking size of ssize_t: "4"
Checking size of dev_t: "4"
Checking size of ino_t: "4"
Checking size of time_t: "4"
Checking size of void*: "4"
Checking for C99 vsnprintf: OK
Checking for HAVE_SECURE_MKSTEMP: OK

The right hand side of each line can be in one of several formats, all of which map to a exit code and a returned string. There are also shortcut forms:

  • a OK means that the test should return a exit code of 0, and an empty string
  • a FAIL or NO means that the test returns a exit code of 255, and an empty string
  • a UNKNOWN means the test has an unknown result
  • a "STRING" or 'STRING' means the test has an exit code of 0, and the given string

The general form is:

Some message: (RETCODE, RETSTRING)

here are some examples:

Some message: (0, "")
Some message: (23, "")
Some message: (0, "a string")
Some message: (1, "another string")

If your configure step completes, but there are still UNKNOWN values in your answers file, then the following message will be displayed:

Cross answers file arm.txt is incomplete

and you will need to continue with the iterative process until all answers are filled in

ABI Checking

The waf build has the ability to check that developers are not changing the ABI of libraries. This is enabled by default if you use configure with the --enable-developer option, or if you use the --abi-check option. You can disable this ABI checking by using --abi-check-disable

The way the ABI checking works is this:

  • A library declaration that uses ABI checking gets two extra options, abi_file and abi_match.
  • The ABI signatures are generated using the buildtools/scripts/abi_gen.sh, which uses nm and gdb to generate a set of ABI signatures
  • when the library is linked, the build system will check if the ABI signatures have changed. If they have, then it will suggest modifying the major version number if a symbol has been changed or removed, and the minor version number if a symbol has been added. The build will fail.
  • if the ABI has changed and the build fails, then you should update the library version (the VERSION variable in the library wscript) and then re-run the build. That will regenerate the new signatures file. This should be committed to the git tree.
  • if the ABI has changed for an existing ABI version (eg. change a symbol) and you are really sure you wish to overwrite the old ABI signatures, then you can build with the --abi-update option. That will cause the new ABI signatures to be written out to the ABI file for the existing ABI version.
  • ABI checking will only be done if your system supports the -fvisibility=hidden compiler option, as otherwise spurious symbols may appear in the ABI signatures

As an example, if you were to change the 2nd argument of talloc_is_parent() to be non-const, you would see an error message like this:

[13/22] abi_check: lib/talloc/bin/default/lib/talloc/libtalloc.so
libtalloc.so: symbol talloc_is_parent has changed - please update major version
        old_signature: int (const void *, const void *)
        new_signature: int (const void *, void *)
Waf: Leaving directory `/home/tridge/samba/git/combined/lib/talloc/bin'
Build failed: ABI for libtalloc.so has changed - please fix library version then build with --abi-update
See http://wiki.samba.org/index.php/Waf#ABI_Checking for more information
make: *** [all] Error 1

in that case you would need to edit VERSION in lib/talloc/wscript, then run "waf --abi-update" to regenerate the ABI signatures.

If you added a new symbol talloc_is_parent2() then you would see this:

[13/22] abi_check: lib/talloc/bin/default/lib/talloc/libtalloc.so
libtalloc.so: symbol talloc_is_parent2 has been added - please mark it _PRIVATE_ or update minor version
       signature: int (const void *, const void *)
Waf: Leaving directory `/home/tridge/samba/git/combined/lib/talloc/bin'
Build failed: ABI for libtalloc.so has changed - please fix library version then build with --abi-update
See http://wiki.samba.org/index.php/Waf#ABI_Checking for more information
make: *** [all] Error 1

in that case you would need to either mark talloc_is_parent2() as being _PRIVATE_, or you could edit VERSION in lib/talloc/wscript, then rebuild to regenerate the ABI signatures.

Here is an example library declaration with ABI checking for talloc:

       bld.SAMBA_LIBRARY('talloc',
                         'talloc.c',
                         deps='replace',
                         abi_file='ABI/talloc-%s.sigs' % VERSION,
                         abi_match='talloc* _talloc*',
                         hide_symbols=True,
                         vnum=VERSION)

this will place the ABI file in ABI/talloc-2.0.2.sigs for the 2.0.2 version of the library. A separate signatures file is generated for each library version. The abi_match option means that only symbols starting with talloc or _talloc will be considered as part of the ABI.

The hide_symbols=True option tells the build to enable the -fvisibility=hidden compiler option (if supported) for building this library. That is used in combination with the _PUBLIC_ and _PRIVATE_ attributes on functions to ensure that symbols are not exposed in libraries when they shouldn't be. The eventual aim will be to enable symbol hiding in all of Samba, but we first need to ensure that all ABI symbols in libraries are marked as _PUBLIC_.

In some cases you want to enable -fvisibility=hidden only when a library is a builtin library. For example, we want to hide symbols from libreplace when it is builtin to another library, but expose them if libreplace is built as a standalone library. To do that use this:

   bld.SAMBA_LIBRARY('replace',
                     source=REPLACE_SOURCE,
                     group='base_libraries',
                     hide_symbols=bld.BUILTIN_LIBRARY('replace'),
                     deps='crypt dl nsl socket')

Packaging with waf

If you are the distribution maintainer of one of the packages dervived from the Samba git tree, then you will probably want to use the following configure options:

waf configure --disable-rpath --bundled-libraries=NONE

The above options will disable use of rpath (for both build and install), and will force the use of all system libraries. That means you will first need to ensure that your system has up to date versions of all the dependent libraries, otherwise the configure will fail.

You may also want to control the number of builtin libraries. See the section on builtin libraries above.

Apart from that, packaging with a waf build is very similar to packaging with a autotools based system.

You may also want to add JOBS=4 or similar to the make command to get build parallelism. The makefile based waf build defaults to one CPU.

How it works

Each project within Samba (tdb, ldb, talloc etc) gets a wscript file. That file can depend on other project wscript files. I've kept the wscript files for each project simple by using a 'waf tool' called 'wafsamba', which is in buildtools/wafsamba. Wafsamba is an add-on for waf that sets up the waf configuration and build commands to follow the same sorts of conventions that we have in the existing Samba4 build system.

If you want to have a look at how this was done, then look in buildtools/wafsamba/. You will notice that I've put a lot of paranoia checking in there, as I've found that the exiting config.mk files for the bulk of Samba4 contain some 'interesting' features, such as dependency loops and missing dependencies.

The waf build rules are very simple, and easy to modify. For example:

Documentation

waf itself is documented in a online book here: http://freehackers.org/~tnagy/wafbook/single.html

The build rules I've added for Samba also need to be documented. I've started this process using pydoctor.

Parallel builds

One nice thing about the use of waf is we gain parallel builds. On a 16 CPU machine a complete clean Samba4 build takes just 35s, including rebuilding all of the IDL files. It takes over 5 minutes on the same machine with the old build system.

Here is a graph of the build time versus -j flag

16-cpus.png

and here is a picture which shows the degrees of parallelism at different points in the build with 4 CPUs

4-way-parallelism.png

TODO

A short list of work items, not complete!

detect lack of gdb on developer build

Waf should error out earlier if ./configure.developer is run without gdb available for the ABI scripts.

detect system changes

It would be nice to detect changing system headers/libraries and give an error at build stage

Rules for yapp, yacc and lex

We need rules to regenerate files with yapp (for pidl) if the grammar has changed.

We need rules to regenerate files with yacc and lex (for heimdal), here we need to only allow trusted combinations of bison and flex.

If the system has this tools we need to have rules for them, otherwise we need a way to give an error if the "source" has changed but we're not able to regenerate.

update: done for yapp

add shared lib support for other compilers

The AIX, IRIX and TRU64 compilers can generate shared libs. We need to support that and test that it works correctly.

add better optimizations options

we should add more control over the use of -O2, -O1 etc in configure

build static libraries

the standalone libraries need to be built as static libraries as well

COMPLETED TODO ITEMS

speedup null build

the null build now takes about 1.6s which is good enough

fix 'waf test'

all s4 tests now pass with the waf build

check each compiler option

for options like --enable-developer we need to check the compiler supports the various options.

The flags should only be used in the build and not within configure checks.

get the waf build working in the build farm

This will test the portability quite well

its passing on some hosts in the build farm. The others are being worked on

get git revision in logs and -V output

bin/samba -V should show the git revision

waf configure --help

waf configure --help should display 'build' as possible command

implement waf install

We need to call the same install scripts we used from the old build, at least initially.

all done now

add --no-bundled-libraries

done, as --bundled-libraries=LIST

implement LIBRARY->LIBRARY dependency reduction

We want object files used in common between libraries that depend on each other to be removed from the parent library, unless the parent library exports the symbols publicly and the child library is a private library

done. Installed size of s4 with waf is now down to about 110M

make waf builds works from subdirectories

You could be able to "cd source4/client" and run waf

done

waf dist support

done. We now build working tarballs for tdb, ldb, talloc, replace, tevent and samba4

Support waf test --gdb

We need to have a replacement for the old 'make gdbtest' and 'make gdbtest-env'.

done

uuid module

The uuid module in scripting/python should only be built if the python version in use is <= 2.4. Newer versions of Python include this module.

done

parse MAKEFLAGS

If someone uses 'make -j N' then that sets MAKEFLAGS in waf. It would be nice to parse that and setup the right number of jobs, for people who want to do parallel builds with 'make' instead of 'waf'

done: we parse -j and -k now

check RPM build of the Samba libs

check that the spec files work with minimal changes

done: working spec files sent to Simo

Add ABI checks to waf dist

When preparing releases make sure ABI/API are checked for breakage so that it can be handled appropriately by either fixing the code or eventually increasing the .so version for libraries marked stable.

done: see ABI Checking above

implement the non-emulator based cross-compile method

See the proposal here: http://lists.samba.org/archive/samba-technical/2010-April/070563.html and here: http://lists.samba.org/archive/samba-technical/2010-April/070564.html

done: see cross-answers section above

detect configure changes

We need to detect configure related changes to wscripts and give an error at build stage.

done: added --enable-auto-reconfigure option. See discussion on list

ldb standalone build install ldb.inst.h

the ldb standalone build installs a 'ldb.inst.h' file rather than a 'ldb.h' file

ldb standalone build needs to install manual pages

the ldb standalone install no longer installs manual pages

detect lack of gdb on developer build

Waf should error out earlier if ./configure.developer is run without gdb available for the ABI scripts.

use of system heimdal

it is not yet possible to use the system-installed heimdal when using the waf build, it will always use the internal heimdal at the moment.

there is some work work in progress in heimdal_build/wscript_configure to support this.


Run autogen-waf.sh within waf dist

Tarballs should come with ./configure already available, run autogen-waf.sh as part of waf dist so that the only thing users need to do is ./configure && make

support public_deps symbol hiding

We need to use the linker scripts to hide non-public symbols in libraries

shared modules are empty

.so files created for modules do not contain any symbols, making them unusable.