Waf

Revision as of 21:17, 23 February 2010 by AndrewTridgell (talk | contribs) (experiments in using waf for Samba)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Experiments with using waf to build Samba

As I mentioned previously, I've been experimenting with using waf for Samba builds. So far it looks very promising. The waf maintainer is being extremely helpful.

I've added 'wscript' waf rules for libreplace, tevent, tdb, talloc and ldb. Configure, build, install all work.

I've set it up so builds appear in bin/ (so if building in source4/lib/ldb then you get source4/lib/ldb/bin/ldbadd for example). This keeps the current workflow we use in Samba development.

rpath is used so the binaries can be run directly (so no stuffing about with LD_LIBRARY_PATH etc for gdb, valgrind and other testing)

all built files (object, libs, binaries) are in bin. The source tree is never modified. If you do "rm -rf bin" then you're back to a pristine tree.

I've also written a script to auto-convert our existing config.mk files for Samba4 to wscript_build scripts. This would allow us to run the two build systems in parallel with very little pain, as you run one script to re-generate all the waf build scripts.

The full Samba4 build is not working yet. I still have quite a bit to do on the rules for pidl, asn1, errtable etc, plus rules for how we will handle library dependencies.

waf is certainly fast. It uses full content hashing for dependencies, but so far that hasn't been a problem. It also has a 'wafcache' which is a ccache like system, but that applies to all builds, so it will safely cache pidl, asn1, libraries, configure checks etc.

The main thing I've learned from trying this is that the build rules for Samba4 are very complex, and do not lend themselves to a simple representation. We have 485 build targets in the Samba4 tree (ie. subsystems, modules, binaries, libraries etc).

Trying it out

This is highly experimental at the moment, but if you do want to give it a go then try this:

  • setup your PATH to point at the buildtools/ directory in the root of the Samba sources
export PATH=/home/tridge/samba-waf/buildtools:$PATH
  • don't install waf on your system, instead use the one in buildtools/. This ensures we are all always using the same version.
  • to enable the wafcache (which speeds things up a lot), do this:
export WAFCACHE=$HOME/.wafcache
mkdir -p $WAFCACHE
  • now try one of the trees that have been converted to waf. For example, to try the tdb tree do this:
cd lib/tdb
waf configure --enable-developer --prefix=$HOME/testprefix
waf
  • that will configure tdb and build the binaries in lib/tdb/bin/. To install them use:
waf install
  • The other trees that have been converted are lib/replace, lib/talloc, lib/tevent and source4/lib/ldb

Other commands

Try "waf --help" for help on using waf. Try "waf configure --help" for the equivalent of "./configure --help". I haven't added all of our Samba configuration options yet, but I've put in a few (such as --enable-developer). See lib/replace/wscript for how these are added.

You may also find "waf -v" useful for seeing the build commands. Or use "waf -v clean build" to clean and build a tree, with the commands shown. Use "waf -p" to show a progress bar on the build.


How it works

Each project within Samba (tdb, ldb, talloc etc) gets a wscript file. That file can depend on other project wscript files. I've kept the wscript files for each project simple by using a 'waf tool' called 'wafsamba.py', which is an add-on for waf that sets up the waf configuration and build commands to follow the same sorts of conventions that we have in the existing Samba4 build system.

If you want to have a look at how this was done, then look in lib/replace/wafsamba.py. You will notice that I've put a lot of paranoia checking in there, as I've found that the exiting config.mk files for the bulk of Samba4 contain some 'interesting' features, such as dependency loops and missing dependencies.

Documentation

waf itself is documented in a online book here: http://freehackers.org/~tnagy/wafbook/single.html

The build rules I've added for Samba would also need to be documented if we adopt waf, probably by annotating wafsamba.py.

Converting Samba4

Converting libreplace, talloc, ldb etc was easy. The hard bit comes when trying to do a full Samba4 build. If you want to see what I've doing for that, have a look in buildtools/mktowscript/. That contains an extremely hackish perl script that converts our existing .mk files to wscript_build files.