Waf: Difference between revisions
No edit summary |
No edit summary |
||
Line 1: | Line 1: | ||
= Using waf to build Samba = |
= Using waf to build Samba = |
||
As of March 2010, the waf build of Samba4 is now |
As of March 2010, the waf build of Samba4 is now complete. We also |
||
have waf build rules for libreplace, talloc, tdb, tevent, ldb and |
have waf build rules for libreplace, talloc, tdb, tevent, ldb and |
||
Samba4. Kai is working on the waf build for Samba3. |
Samba4. Kai is working on the waf build for Samba3. |
||
Main features: |
|||
The waf builds are set up so builds appear in bin/ (so if building in |
|||
source4/lib/ldb then you get source4/lib/ldb/bin/ldbadd for |
|||
example). This is consistent with the current workflow we use in Samba |
|||
development. |
|||
* rpath is used so the binaries can be run directly |
|||
* building and configuring is fast |
|||
without the need for LD_LIBRARY_PATH. If rpath is not supported then |
|||
* full dependencies are checked, so much less need for make clean |
|||
LD_LIBRARY_PATH is automatically set for running the testsuites. |
|||
* wrappers are provided for ./configure and make |
|||
* builds are much smaller (around 110M versus 1.1G for the old Samba4 build) |
|||
* project rules are checked to ensure no object duplication |
|||
* supports creation of working tarballs for our standalone libraries |
|||
* builds on more build-farm hosts than the existing build system |
|||
* much cleaner build rules (in wscript files). Should be easy to learn |
|||
== Getting the source == |
|||
All built files (object, libs, binaries) are in bin. The source tree |
|||
is never modified (with one minor exception for gen_ndr, as described |
|||
below). If you do "rm -rf bin" then you're back to a pristine tree. |
|||
waf is certainly fast. It uses full content hashing for dependencies, |
|||
but still manages to run quickly. It also has a 'wafcache' which is a |
|||
ccache like system, but that applies to all builds, so it will safely |
|||
cache pidl, asn1, libraries, configure checks etc. |
|||
== Getting the code == |
|||
You need to do a git checkout of the waf-wip branch like this: |
You need to do a git checkout of the waf-wip branch like this: |
||
Line 38: | Line 32: | ||
== Trying it out == |
== Trying it out == |
||
There are two ways to use the waf build. You can use |
There are two ways to use the waf build. You can either use the |
||
configure wrappers, or you can call waf directly. |
|||
waf. |
|||
=== Using the configure wrappers === |
|||
To build Samba4 using the configure wrappers do this: |
|||
./autogen-waf.sh |
|||
./configure --normal-configure-options |
|||
make |
|||
make test |
|||
You can also use ./configure.developer as usual. |
|||
If you |
If you use the configure wrappers then note that the makefile is setup |
||
with "JOBS=1" by default. To take advantage of parallel builds, use |
|||
such as a progress bar with 'waf -p' and the ability to specify which |
|||
something like: |
|||
tests to run in the testsuite. To run waf directly do this: |
|||
make JOBS=4 |
|||
* setup your PATH to point at the buildtools/bin directory in the root of the Samba sources |
|||
=== Using waf directly === |
|||
You will probably find it useful to be able to call waf directly, as |
|||
this offers additional features. To do that follow these steps: |
|||
* setup your PATH to point at the buildtools/bin directory |
|||
export PATH=$HOME/samba-waf/buildtools/bin:$PATH |
export PATH=$HOME/samba-waf/buildtools/bin:$PATH |
||
Line 56: | Line 67: | ||
example, to try the tdb tree do this: |
example, to try the tdb tree do this: |
||
cd |
cd source4 |
||
waf configure --enable-developer --prefix=$HOME/testprefix |
waf configure --enable-developer --prefix=$HOME/testprefix |
||
waf |
waf |
||
* that will configure tdb and build the binaries in lib/tdb/bin/. To install them use: |
* that will configure tdb and build the binaries in lib/tdb/bin/. To install them use: |
||
Line 66: | Line 77: | ||
* The other trees that have been converted are lib/replace, |
* The other trees that have been converted are lib/replace, |
||
lib/talloc, lib/tevent and source4/lib/ldb and source4 |
lib/talloc, lib/tevent and source4/lib/ldb and source4 |
||
== Building Samba4 == |
|||
The current waf-wip branch can build Samba4. This is still a work in |
|||
progress, but if you want to try it, then try this: |
|||
cd samba-waf/source4 |
|||
./autogen-waf.sh |
|||
./configure.developer |
|||
make |
|||
make quicktest |
|||
This avoids waf commands directly. To get the most out of waf you |
|||
should use waf itself instead. Have a look at the configure script and |
|||
Makefile that autogen-waf.sh produces to see examples of how to use |
|||
it. |
|||
== Other commands == |
== Other commands == |
||
Line 87: | Line 82: | ||
Try "waf --help" for help on using waf. Try "waf configure --help" for |
Try "waf --help" for help on using waf. Try "waf configure --help" for |
||
the equivalent of "./configure --help". I haven't added all of our |
the equivalent of "./configure --help". I haven't added all of our |
||
Samba configuration options yet, but I've put in |
Samba configuration options yet, but I've put in the most commonly |
||
--enable-developer). |
used ones (such as --enable-developer). |
||
You may also find "waf -v" useful for seeing the build commands. Or |
You may also find "waf -v" useful for seeing the build commands. Or |
||
Line 98: | Line 93: | ||
waf configure --enable-developer -b devbuild |
waf configure --enable-developer -b devbuild |
||
While will set you up for creating the binaries in devbuild instead of |
|||
the default of bin/. |
|||
If you want to debug why something isn't working, then you can either |
If you want to debug why something isn't working, then you can either |
||
increase the verbosity of waf with something like "waf -vvv", or you |
|||
can enable tracing of a particular phase of the build. For example, |
|||
you could trace the build dependencies like this: |
|||
waf -v --zone=deps |
waf -v --zone=deps |
||
It is sometimes useful to disable the wafcache when debugging. use the --nocache option for that: |
|||
waf -v --nocache clean build |
|||
== How it works == |
== How it works == |
||
Line 129: | Line 124: | ||
* the rules for libreplace, which include most of our configure checks, are here [http://git.samba.org/?p=tridge/samba.git;a=blob;f=lib/replace/wscript;hb=waf-wip lib/replace/wscript] |
* the rules for libreplace, which include most of our configure checks, are here [http://git.samba.org/?p=tridge/samba.git;a=blob;f=lib/replace/wscript;hb=waf-wip lib/replace/wscript] |
||
* the 'wafsamba' waf tool which allows us to describe samba build rules succinctly are here [http://git.samba.org/?p=tridge/samba.git;a=blob;f=buildtools/wafsamba/wafsamba.py;hb=waf-wip buildtools/wafsamba/wafsamba.py] |
* the 'wafsamba' waf tool which allows us to describe samba build rules succinctly are here [http://git.samba.org/?p=tridge/samba.git;a=blob;f=buildtools/wafsamba/wafsamba.py;hb=waf-wip buildtools/wafsamba/wafsamba.py] |
||
== Adding new commands == |
|||
In Samba we are used to a number of non-standard make targets, such as |
|||
'make quicktest'. We can add the same targets using waf. Even better, |
|||
when you add a new target in waf it shows up in 'waf --help'. |
|||
Main commands (example: ./waf build -j4) |
|||
clean : removes the build files |
|||
configure: configures the project |
|||
dist : makes a tarball for redistributing the sources |
|||
distcheck: checks if the sources compile (tarball from 'dist') |
|||
distclean: removes the build directory |
|||
install : installs the build files |
|||
quicktest: Run the quick test suite |
|||
test : Run the full test suite |
|||
uninstall: removes the installed files |
|||
The commands themselves are arbitrary python. See the source4/selftest |
|||
directory for some examples on how this works. |
|||
== Documentation == |
== Documentation == |
||
Line 156: | Line 130: | ||
here: http://freehackers.org/~tnagy/wafbook/single.html |
here: http://freehackers.org/~tnagy/wafbook/single.html |
||
The build rules I've added for Samba |
The build rules I've added for Samba also need to be documented. I've |
||
started this process using pydoctor. |
|||
if we adopt waf, probably by using the python documentation tools on |
|||
the wafsamba tool. |
|||
= Parallel builds = |
= Parallel builds = |
||
One nice thing about the use of waf is we gain parallel builds. On a |
One nice thing about the use of waf is we gain parallel builds. On a |
||
Samba4 build takes just 35s, including |
16 CPU machine a complete clean Samba4 build takes just 35s, including |
||
rebuilding all of the IDL files. It takes over 5 minutes on the same |
|||
the same machine with the old build system. |
|||
machine with the old build system. |
|||
Here is a graph of the build time versus -j flag |
Here is a graph of the build time versus -j flag |
||
Line 176: | Line 149: | ||
A short list of work items, not complete! |
A short list of work items, not complete! |
||
== detect system changes == |
|||
== make waf builds works from subdirectories == |
|||
You could be able to "cd source4/client" and run waf |
|||
== reliable builds == |
|||
we need to detect changing system headers/libraries and give an error. |
|||
we should include system headers in the dependecy calculation. |
|||
It would be nice to detect changing system headers/libraries and give an error. |
|||
Note that this TODO is well beyond what the current build system can |
|||
do. We're aiming for a very high standard of dependency checking. |
|||
== Rules for yapp, yacc and lex == |
== Rules for yapp, yacc and lex == |
||
Line 252: | Line 216: | ||
done. Installed size of s4 with waf is now down to about 110M |
done. Installed size of s4 with waf is now down to about 110M |
||
== make waf builds works from subdirectories == |
|||
You could be able to "cd source4/client" and run waf |
|||
done |
|||
== waf dist support == |
|||
done. We now build working tarballs for tdb, ldb, talloc, replace, |
|||
tevent and samba4 |
Revision as of 08:56, 4 April 2010
Using waf to build Samba
As of March 2010, the waf build of Samba4 is now complete. We also have waf build rules for libreplace, talloc, tdb, tevent, ldb and Samba4. Kai is working on the waf build for Samba3.
Main features:
- rpath is used so the binaries can be run directly
- building and configuring is fast
- full dependencies are checked, so much less need for make clean
- wrappers are provided for ./configure and make
- builds are much smaller (around 110M versus 1.1G for the old Samba4 build)
- project rules are checked to ensure no object duplication
- supports creation of working tarballs for our standalone libraries
- builds on more build-farm hosts than the existing build system
- much cleaner build rules (in wscript files). Should be easy to learn
Getting the source
You need to do a git checkout of the waf-wip branch like this:
git clone git://git.samba.org/tridge/samba.git samba-waf cd samba-waf git checkout -b waf-wip origin/waf-wip
To update your checkout when something has changed, run this:
cd samba-waf git pull --rebase
Trying it out
There are two ways to use the waf build. You can either use the configure wrappers, or you can call waf directly.
Using the configure wrappers
To build Samba4 using the configure wrappers do this:
./autogen-waf.sh ./configure --normal-configure-options make make test
You can also use ./configure.developer as usual.
If you use the configure wrappers then note that the makefile is setup with "JOBS=1" by default. To take advantage of parallel builds, use something like:
make JOBS=4
Using waf directly
You will probably find it useful to be able to call waf directly, as this offers additional features. To do that follow these steps:
- setup your PATH to point at the buildtools/bin directory
export PATH=$HOME/samba-waf/buildtools/bin:$PATH
- don't install waf on your system, instead use the one in
buildtools/. This ensures we are all always using the same version.
- now try one of the trees that have been converted to waf. For
example, to try the tdb tree do this:
cd source4 waf configure --enable-developer --prefix=$HOME/testprefix waf
- that will configure tdb and build the binaries in lib/tdb/bin/. To install them use:
waf install
- The other trees that have been converted are lib/replace,
lib/talloc, lib/tevent and source4/lib/ldb and source4
Other commands
Try "waf --help" for help on using waf. Try "waf configure --help" for the equivalent of "./configure --help". I haven't added all of our Samba configuration options yet, but I've put in the most commonly used ones (such as --enable-developer).
You may also find "waf -v" useful for seeing the build commands. Or use "waf -v clean build" to clean and build a tree, with the commands shown. Use "waf -p" to show a progress bar on the build.
This command illustrates the use of multiple build directories:
waf configure --enable-developer -b devbuild
While will set you up for creating the binaries in devbuild instead of the default of bin/.
If you want to debug why something isn't working, then you can either increase the verbosity of waf with something like "waf -vvv", or you can enable tracing of a particular phase of the build. For example, you could trace the build dependencies like this:
waf -v --zone=deps
How it works
Each project within Samba (tdb, ldb, talloc etc) gets a wscript file. That file can depend on other project wscript files. I've kept the wscript files for each project simple by using a 'waf tool' called 'wafsamba', which is in buildtools/wafsamba. Wafsamba is an add-on for waf that sets up the waf configuration and build commands to follow the same sorts of conventions that we have in the existing Samba4 build system.
If you want to have a look at how this was done, then look in buildtools/wafsamba/. You will notice that I've put a lot of paranoia checking in there, as I've found that the exiting config.mk files for the bulk of Samba4 contain some 'interesting' features, such as dependency loops and missing dependencies.
The waf build rules are very simple, and easy to modify. For example:
- the rules for building tdb are here lib/tdb/wscript
- the rules for ldb are here source4/lib/ldb/wscript
- the rules for libreplace, which include most of our configure checks, are here lib/replace/wscript
- the 'wafsamba' waf tool which allows us to describe samba build rules succinctly are here buildtools/wafsamba/wafsamba.py
Documentation
waf itself is documented in a online book here: http://freehackers.org/~tnagy/wafbook/single.html
The build rules I've added for Samba also need to be documented. I've started this process using pydoctor.
Parallel builds
One nice thing about the use of waf is we gain parallel builds. On a 16 CPU machine a complete clean Samba4 build takes just 35s, including rebuilding all of the IDL files. It takes over 5 minutes on the same machine with the old build system.
Here is a graph of the build time versus -j flag
TODO
A short list of work items, not complete!
detect system changes
It would be nice to detect changing system headers/libraries and give an error.
Rules for yapp, yacc and lex
We need rules to regenerate files with yapp (for pidl) if the grammar has changed.
We need rules to regenerate files with yacc and lex (for heimdal), here we need to only allow trusted combinations of bison and flex.
If the system has this tools we need to have rules for them, otherwise we need a way to give an error if the "source" has changed but we're not able to regenerate.
update: done for yapp
COMPLETED TODO ITEMS
speedup null build
the null build now takes about 1.6s which is good enough
fix 'waf test'
all s4 tests now pass with the waf build
check each compiler option
for options like --enable-developer we need to check the compiler supports the various options.
The flags should only be used in the build and not within configure checks.
get the waf build working in the build farm
This will test the portability quite well
its passing on some hosts in the build farm. The others are being worked on
get git revision in logs and -V output
bin/samba -V should show the git revsion
waf configure --help
waf configure --help should display 'build' as possible command
implement waf install
We need to call the same install scripts we used from the old build, at least initially.
all done now
add --no-bundled-libraries
done, as --bundled-libraries=LIST
implement LIBRARY->LIBRARY dependency reduction
We want object files used in common between libraries that depend on each other to be removed from the parent library, unless the parent library exports the symbols publicly and the child library is a private library
done. Installed size of s4 with waf is now down to about 110M
make waf builds works from subdirectories
You could be able to "cd source4/client" and run waf
done
waf dist support
done. We now build working tarballs for tdb, ldb, talloc, replace, tevent and samba4