Messaging

Multiple smbd (and nmbd, winbindd) processes usually run simultaneously - a main daemon process and couple of connections processes. Samba code has special API that allows different processes send messages each other. There are two implementations of the messaging code - different in current Samba3 and Samba4 code.

TDB implementation (samba3 and trunk)

This implementaion uses TDB file for storing messages and UNIX signals for notifying message receiver.

All messages are stored in messages.tdb file (one can find it in var/locks directory). Each message is just a database record that has six fields:

  • messaging version (there is only one version now - 1)
  • message type
  • sender process id
  • receiver process id
  • message data length
  • message data

There are lot of message types in Samba3, you can see them all in include/messages.h file.

Process sends a message to other process with special function message_send_pid and to all processes with message_send_all. Each process should call message_dispatch routinely to check if messages were received and run appropriate handler.

Messaging-tdb.png

(you can get image source here - [1])

UNIX datagram sockets implementation (samba4)

This implementation uses UNIX domain sockets. Each smbd listens on a unix domain (datagram) socket with the name based on its pid. All sockets are located in var/locks directory. One message is packed into one datagram, so now we have message size problem - there is no way to send messages with size larger than allowed by a datagram.

Messaging-socket.png

You can get the patch with this messaging approach for samba3 - [2]

This implementation depends on net.unix.max_dgram_qlen sysctl variable - it's the maximum datagram queue length. You should increase the variable's value if your messaging under high pressure (e.g. if testing it with torture tests).

UNIX stream sockets implementation

This implementation uses UNIX domain sockets too, but stream sockets. Each smbd listens on a unix domain (stream) socket with the name based on its pid. All sockets are located in var/locks directory. When a message is about to be sent, the sender connect(2)s to that socket and sends the whole message. As stream sockets are bidirectional, the response can use the same socket. This could be implemented by a message socket cache that holds for example the last N (e.g. 10) connections. All data transfering is being done asynchronousy with main a select loop (message_dispatch does all connections-related work).

Messaging-stream-socket.png

You can get the patch with this messaging approach for samba3 - [3]

Testing and benchmarking

Messaging speed can be benchmarked with LOCAL-MESSAGING test from Samba4 smbtorture. There is a backport of this test to samba-3 (and trunk) messaging code - torture/msgtest.c (see current svn branches).

Implementations stated above were compared with this tool (average results based on 5 tests, -O2 compilation, Thinkpad T41 1700 MHz hardware):

  • TDB ~ 64850 msg/s (100%)
  • TDB (use mmap = no) ~ 34220 msg/s (53%)
  • datagram sockets ~ 58400 msg/s (90%)
  • stream sockets ~ 16850 msg/s (26%)

Overall bechmarking can be done with dbench tool. Here are results of running dbench with 10 clients:

  • TDB ~ 7.65 Mb/s (100%)
  • datagram sockets ~ 7.26 Mb/s (95%)
  • stream sockets ~ 6.78 Mb/s (90%)

and with 5 clients:

  • TDB ~ 8.3 Mb/s (100%)
  • datagram sockets ~ 7.8 Mb/s (94%)
  • stream sockets ~ 7.3 Mb/s (88%)

All results are very rough, but they can be used for a compare.

General messaging code

New messaging features are tested in vl-messaging branch.

All approaches listed above were modularized and combined in the single patch to current Samba3 branch: [4]. With it one can select messaging type at compile time and runtime:

There is new configure option --with-messaging=, with it a comma-separated list of wanted messaging types (any of: tdb, dgram, stream) should be used. At run time any of compiled in messaging types can be selected with messaging type configuration variable (also: tdb, dgram, stream).

Messaging modules hierarchy are shown on the diagram:

Messaging-modules-hierarchy.png

In this implementation messaging subsystem should be initialized with message_init and freed with message_end.