Waf: Difference between revisions

From SambaWiki
No edit summary
No edit summary
Line 184: Line 184:
system.
system.


waf configure --bundled-libraries=!tdb,ALL

That means to bundle all libraries except tdb. In that case tdb comes
from the system.

The bundled library checks also include a library dependency check. It
is considered an error to get a library from the system, but to get a
dependency of that library as a bundled library. To give an example of
why that matters, consider this configure command:

waf configure --bundled-libraries=!tevent,ALL

That would tell configure to bundle all libraries except tevent. The
problem with this is that tevent depends on talloc. So if we did used
the above settings, then we would end up linking in talloc twice, once
from the system libtevent.so, and once from the bundled
libtalloc-samba4.so.

The configure process detects this error, and gives an error.

==== configure --builtin-libraries=LIST ====

The --builtin-libraries configure option allows you to control what
libraries are directly (statically) linked into the program, instead
of being linked as shared libraries.

You might, for example, want to link libreplace directly into Samba4,
in which case you would use:

waf configure --builtin-libraries=replace

For Samba4 the default is to build all libraries as shared
libraries. The default for standalone builds of tdb, talloc, tevent
and ldb are to build libreplace as a builtin, with the other
dependent libraries built as shared libraries.

All of this is controlled in the wscript files with statements like
this:

if conf.CHECK_BUNDLED_SYSTEM('tevent', minversion=VERSION,
onlyif='talloc', implied_deps='replace talloc'):
conf.define('USING_SYSTEM_TEVENT', 1)

that says that when building a standalone tevent, we should only use
the system version of tevent if we also have the system version of
talloc.

==== configure --bundled-library-extension ====

The --bundled-library-extension option allows you to control the extra
extension added to the names of bundled libraries. For example, if you
wanted bundled libraries to be called (for example) libldb-s4.so, then
use:

waf configure --bundled-library-extension=s4

You can also provide a list of libraries not to apply the extension
to. For example:

waf configure --bundled-extension-exception=talloc

would tell configure not to add a bundling extension when building
talloc.





Revision as of 09:45, 4 April 2010

Using waf to build Samba

As of March 2010, the waf build of Samba4 is now complete. We also have waf build rules for libreplace, talloc, tdb, tevent, ldb and Samba4. Kai is working on the waf build for Samba3.

Main features:

  • rpath is used so the binaries can be run directly
  • building and configuring is fast
  • full dependencies are checked, so much less need for make clean
  • wrappers are provided for ./configure and make
  • builds are much smaller (around 110M versus 1.1G for the old Samba4 build)
  • project rules are checked to ensure no object duplication
  • supports creation of working tarballs for our standalone libraries
  • builds on more build-farm hosts than the existing build system
  • much cleaner build rules (in wscript files). Should be easy to learn

Getting the source

You need to do a git checkout of the waf-wip branch like this:

git clone git://git.samba.org/tridge/samba.git samba-waf
cd samba-waf
git checkout -b waf-wip origin/waf-wip

To update your checkout when something has changed, run this:

cd samba-waf
git pull --rebase

Trying it out

There are two ways to use the waf build. You can either use the configure wrappers, or you can call waf directly.

Using the configure wrappers

To build Samba4 using the configure wrappers do this:

./autogen-waf.sh
./configure --normal-configure-options
make
make test

You can also use ./configure.developer as usual.

If you use the configure wrappers then note that the makefile is setup with "JOBS=1" by default. To take advantage of parallel builds, use something like:

make JOBS=4

Using waf directly

You will probably find it useful to be able to call waf directly, as this offers additional features. To do that follow these steps:

  • setup your PATH to point at the buildtools/bin directory
export PATH=$HOME/samba-waf/buildtools/bin:$PATH
  • don't install waf on your system, instead use the one in
 buildtools/. This ensures we are all always using the same version.
  • now try one of the trees that have been converted to waf. For
 example, to try the tdb tree do this:
cd source4
waf configure --enable-developer --prefix=$HOME/testprefix
waf
  • that will configure tdb and build the binaries in lib/tdb/bin/. To install them use:
waf install
  • The other trees that have been converted are lib/replace,
 lib/talloc, lib/tevent and source4/lib/ldb and source4

Other commands

Try "waf --help" for help on using waf. Try "waf configure --help" for the equivalent of "./configure --help". I haven't added all of our Samba configuration options yet, but I've put in the most commonly used ones (such as --enable-developer).

You may also find "waf -v" useful for seeing the build commands. Or use "waf -v clean build" to clean and build a tree, with the commands shown. Use "waf -p" to show a progress bar on the build.

This command illustrates the use of multiple build directories:

waf configure --enable-developer -b devbuild

While will set you up for creating the binaries in devbuild instead of the default of bin/.

If you want to debug why something isn't working, then you can either increase the verbosity of waf with something like "waf -vvv", or you can enable tracing of a particular phase of the build. For example, you could trace the build dependencies like this:

 waf -v --zone=deps

Developing with waf

With the waf based build we get a number of new features that we didn't have in the old build system.

creating distributon tarballs: waf dist

The waf build automates the process of creating distribution tarballs. It also automatically bundles the needed build tools and the library dependencies. For example:

cd lib/tevent
waf dist

Will create tevent-0.9.8.tar.gz. If you unpack that tarball then you'll find it has automatically included the talloc and libreplace libraries in the distribution. It has also setup a autogen-waf.sh which will generate a working configure script and makefile.

working with libraries

One of the main challenges we had with the previous build system was how we built and used shared libraries. We had difficulty getting a system in place that worked well both for the package maintainers in distributions, and for people building Samba themselves.

To address these problems the waf build uses a number of strategies:

  • rpath is used to make the lives of developers easier
  • 'library bundling' is used to ensure libraries don't conflict with
 system libraries
  • configure options are provided to customise the library handling

Perhaps the easiest way to explain the new system is by explaining each of the key configure options, and give common use cases.

configure --bundled-libraries=LIST

The --bundled-libraries option allows you to choose which libraries are 'bundled' with the build. A bundled library is a in-tree library which may have a system equivalent on some systems. The waf build process needs to decide for each library whether it will bundle it or not.

If the configure process does decide to bundle a library, then it will automatically append a suffix onto the library name. The default suffix is the name of the package you are building. For example, if when building Samba4 you bundle the tdb library, then it will build a libtdb-samba4.so shared library.

The reason for the renaming is that it allows the libraries to be installed safely. If we left the name as libtdb.so, then there would be no way (without rpath) to run the newly built tdb library for Samba, but leave the rest of the system using the system supplied tdb library.

The default is to try to use system libraries if they are at least the same version as the in-tree version. This means that if we increase the version number of our in-tree copy of tdb, that we will prefer that in-tree copy as a bundled library until the system version catches up.

The following examples show how to control this process:

waf configure --bundled-libraries=NONE

The 'NONE' option is what many distribution packagers will choose. It tells waf not to bundle any libraries. That means that if the configure process can't find a suitable system library (of the right version) the configure will fail.

waf configure --bundled-libraries=ALL

The 'ALL' option means to bundle all libraries. That means any equivalent system libraries for our in-tree libraries are ignored.

waf configure --bundled-libraries=ldb,NONE

That means to bundle ldb, but to get all other libraries from the system.

waf configure --bundled-libraries=!tdb,ALL

That means to bundle all libraries except tdb. In that case tdb comes from the system.

The bundled library checks also include a library dependency check. It is considered an error to get a library from the system, but to get a dependency of that library as a bundled library. To give an example of why that matters, consider this configure command:

waf configure --bundled-libraries=!tevent,ALL

That would tell configure to bundle all libraries except tevent. The problem with this is that tevent depends on talloc. So if we did used the above settings, then we would end up linking in talloc twice, once from the system libtevent.so, and once from the bundled libtalloc-samba4.so.

The configure process detects this error, and gives an error.

configure --builtin-libraries=LIST

The --builtin-libraries configure option allows you to control what libraries are directly (statically) linked into the program, instead of being linked as shared libraries.

You might, for example, want to link libreplace directly into Samba4, in which case you would use:

waf configure --builtin-libraries=replace

For Samba4 the default is to build all libraries as shared libraries. The default for standalone builds of tdb, talloc, tevent and ldb are to build libreplace as a builtin, with the other dependent libraries built as shared libraries.

All of this is controlled in the wscript files with statements like this:

   if conf.CHECK_BUNDLED_SYSTEM('tevent', minversion=VERSION,
                                onlyif='talloc', implied_deps='replace talloc'):
       conf.define('USING_SYSTEM_TEVENT', 1)

that says that when building a standalone tevent, we should only use the system version of tevent if we also have the system version of talloc.

configure --bundled-library-extension

The --bundled-library-extension option allows you to control the extra extension added to the names of bundled libraries. For example, if you wanted bundled libraries to be called (for example) libldb-s4.so, then use:

waf configure --bundled-library-extension=s4

You can also provide a list of libraries not to apply the extension to. For example:

waf configure --bundled-extension-exception=talloc

would tell configure not to add a bundling extension when building talloc.


How it works

Each project within Samba (tdb, ldb, talloc etc) gets a wscript file. That file can depend on other project wscript files. I've kept the wscript files for each project simple by using a 'waf tool' called 'wafsamba', which is in buildtools/wafsamba. Wafsamba is an add-on for waf that sets up the waf configuration and build commands to follow the same sorts of conventions that we have in the existing Samba4 build system.

If you want to have a look at how this was done, then look in buildtools/wafsamba/. You will notice that I've put a lot of paranoia checking in there, as I've found that the exiting config.mk files for the bulk of Samba4 contain some 'interesting' features, such as dependency loops and missing dependencies.

The waf build rules are very simple, and easy to modify. For example:

Documentation

waf itself is documented in a online book here: http://freehackers.org/~tnagy/wafbook/single.html

The build rules I've added for Samba also need to be documented. I've started this process using pydoctor.

Parallel builds

One nice thing about the use of waf is we gain parallel builds. On a 16 CPU machine a complete clean Samba4 build takes just 35s, including rebuilding all of the IDL files. It takes over 5 minutes on the same machine with the old build system.

Here is a graph of the build time versus -j flag

16-cpus.png


TODO

A short list of work items, not complete!

detect system changes

It would be nice to detect changing system headers/libraries and give an error.

Rules for yapp, yacc and lex

We need rules to regenerate files with yapp (for pidl) if the grammar has changed.

We need rules to regenerate files with yacc and lex (for heimdal), here we need to only allow trusted combinations of bison and flex.

If the system has this tools we need to have rules for them, otherwise we need a way to give an error if the "source" has changed but we're not able to regenerate.

update: done for yapp

COMPLETED TODO ITEMS

speedup null build

the null build now takes about 1.6s which is good enough

fix 'waf test'

all s4 tests now pass with the waf build

check each compiler option

for options like --enable-developer we need to check the compiler supports the various options.

The flags should only be used in the build and not within configure checks.

get the waf build working in the build farm

This will test the portability quite well

its passing on some hosts in the build farm. The others are being worked on

get git revision in logs and -V output

bin/samba -V should show the git revsion

waf configure --help

waf configure --help should display 'build' as possible command

implement waf install

We need to call the same install scripts we used from the old build, at least initially.

all done now

add --no-bundled-libraries

done, as --bundled-libraries=LIST

implement LIBRARY->LIBRARY dependency reduction

We want object files used in common between libraries that depend on each other to be removed from the parent library, unless the parent library exports the symbols publicly and the child library is a private library

done. Installed size of s4 with waf is now down to about 110M

make waf builds works from subdirectories

You could be able to "cd source4/client" and run waf

done

waf dist support

done. We now build working tarballs for tdb, ldb, talloc, replace, tevent and samba4