Samba4/s3fs: Difference between revisions

From SambaWiki
(a new page explaining s3fs)
 
m (/* update, note: this needs more work)
 
(27 intermediate revisions by 7 users not shown)
Line 1: Line 1:
=What is s3fs=
=What is s3fs=
s3fs is the name that has been given to a development effort to make possible the agreed default file server configuration for Samba 4.0
s3fs is the name that was given to the development effort to make possible the agreed default file server configuration for Samba 4.0 as an AD Domain controller.


It was agreed at SambaXP 2010 that Samba 4.0 would release with the smbd file server in use by default, so that users upgrading from Samba 3.x DC environments would still have access to all the features of those environment that they had come to expect.
It was agreed at SambaXP 2010 that Samba 4.0 would release with the smbd file server in use by default, so that users upgrading from Samba 3.x DC environments to the 4.0 AD DC would still have access to all the features of the smbd file server environment that they had come to expect and rely on.

==Why is this called s3fs?==
The primary additional feature that s3fs gives our users is access to the SMB3 protocol developed in smbd. Additionally, this name communicates that this is about using the smbd file server from the Samba 3.x release stream.

For a number of years, two file servers were developed in parallel in the Samba project. The Samba 3.x releases produced the '''smbd''' file server, while the Samba 4.x alpha series had the '''ntvfs''' file server. The ntvfs server never gained more than early SMB2 support. Therefore, '''s3fs''' simply indicates that, integrated into the '''samba''' process used by the AD DC is the file server refined in the Samba 3.x release series and supporting SMB3.

It was hoped that this detail and name would not need to be exposed to users by the time that Samba 4.x was released, while we tried to refine the development and implementation details of this process.


=How it is implemented=
=How it is implemented=
Becuase the smbd file server has a distinct history from the ntvfs file server that the Samba4 development project has used so far, it did not natrually use the same security subsystems and other resources common the the rest of the AD server. Indeed, at a time before the [Franky] effort was started, it was not even in the same GIT tree, and certainly not the same build tree.
Because the smbd file server has a distinct history from the ntvfs file server that the Samba4 development project has used so far, it did not naturally use the same security subsystems and other resources common the the rest of the AD server. Indeed, at a time before the [[Franky]] effort was started, it was not even in the same GIT tree, and certainly not the same build tree.


==The merged build==
==The merged build==
The merged or top level build is the [Waf] build system. Running ./configure and make at the top level of the source tree will build all of Samba, including the previously [Samba3] components. As all duplicate symbol names have been merged, renamed or otherwise dealt with, the merged build has a large number of shared libraries that the whole project builds on, particuarly the IDL genreated file built using [Pidl|PIDL].
The merged or top level build is the [[Waf]] build system. Running ./configure and make at the top level of the source tree will build all of Samba, including the previously [[Samba3]] components. As all duplicate symbol names have been merged, renamed or otherwise dealt with, the merged build has a large number of shared libraries that the whole project builds on, particularly the IDL generated file built using [[Pidl|PIDL]].


==Common structures==
==Common structures==
Key structures describing the security, authentication and authorization state were changed to be in common. In particuar, key structures like *struct auth_session_info* and *struct security_token* now describe the authentication and authorization state across the whole project.
Key structures describing the security, authentication and authorization state were changed to be in common. In particular, key structures like '''struct auth_session_info''' and '''struct security_token''' now describe the authentication and authorization state across the whole project.


==Common subsystems==
==Common subsystems==
As part of the process of building s3fs, a large number of subsystems were made common, including in particular GENSEC. smbd now uses gensec for all 'blob-based' server-side authentication, in all protocols, which has made it possible to both move to a proper implementation of SPNEGO based on GSSAPI, and allowed the activated plugins to be easily switched for AD DC operation.
As part of the process of building s3fs, a large number of subsystems were made common, including in particular GENSEC. smbd now uses gensec for all 'blob-based' server-side authentication, in all protocols, which has made it possible to both move to a proper implementation of SPNEGO based on GSSAPI, and allowed the activated plugins to be easily switched for AD DC operation.

==Named pipe forwarding==
DCE/RPC servers in Samba have in the past been implemented in-process, simply being additional library code that implements the required service, within the forked smbd child process. However, for Samba 4.0 as an AD DC, there are technical reasons why the DCE/RPC servers need to be implemented in a long-lived manner. To communicate between the '''smbd''' process that handles file sharing and the DCE/RPC server, all the SMB named pipe operations are converted into operations on a unix domain socket. For AD services, other parts of the '''samba''' binary then listen on these private unix domain socket connections for SMB named pipe requests.

This also has the advantage that the AD services from the Samba4 heritage do not need to directly link into the '''smbd''' file server or the associated DCE/RPC stack. This approach was originally developed for the [[Franky]] effort for this reason, and remains key to how DCE/RPC is handled in Samba 4.0.

Other DCE/RPC services, such as '''spoolss''' are not forwarded, or are forwarded to another part of the '''smbd''' binary, allowing printing to function on an AD DC.

===smb.conf configuration===
The automatically generated '''fileserver.conf''' has lines such as:
rpc_server:samr = external

to indicate that these pipes are handled outside the '''smbd''' binary.


==Plugins==
==Plugins==
Line 23: Line 43:


====prepare_gensec====
====prepare_gensec====
prepare_gensec returns a *struct gensec_security* that is pre-initialised to use the GENSEC modules that the rest of the AD server uses, including the authentication contexts etc required to authenticate against the AD directory in [LDB|LDBIntro]. When the auth_samba4 module is not in use, the fallback is to the default authentication path, also expressed as a series of GENSEC modules.
prepare_gensec returns a '''struct gensec_security''' that is pre-initialised to use the GENSEC modules that the rest of the AD server uses, including the authentication contexts etc required to authenticate against the AD directory in [[LDB|LDB]]. When the auth_samba4 module is not in use, the fallback is to the default authentication path, also expressed as a series of GENSEC modules.


====make_auth4_context====
====make_auth4_context====
make_auth4_context returns a *struct auth4_context* that similarly operates like the rest of the AD server, for SessionSetupAndX calls that do not use SPNEGO or NTLMSSP.
make_auth4_context returns a '''struct auth4_context''' that similarly operates like the rest of the AD server, for SessionSetupAndX calls that do not use SPNEGO or NTLMSSP.


===pdb_samba4===
===pdb_samba4===
For operation of *smbpasswd*, *pdbedit* and *net sam* tools, and to assist in the migration of Samba 3.x DC environments to Samba 4.0 AD, a passdb module was written based on pdb_ads from [Franky]. This module differs in that it directly calls ldb and the associated modules, allowing offline operation when the *samba* server process is not running. This is particuarly critical for the [samba-tool domain samba3upgrade\Samba4/samba3upgrade/HOWTO] tool.
For operation of '''smbpasswd''', '''pdbedit''' and '''net sam''' tools, and to assist in the migration of Samba 3.x DC environments to Samba 4.0 AD, a passdb module was written based on pdb_ads from [[Franky]]. This module differs in that it directly calls ldb and the associated modules, allowing offline operation when the '''samba''' server process is not running. This is particularly critical for the [[Migrating_a_Samba_NT4_Domain_to_Samba_AD_(Classic_Upgrade)|samba-tool domain samba3upgrade]] tool.


===vfs_samba4===
===vfs_samba4===
Line 38: Line 58:


==Winbind==
==Winbind==
Currently in s3fs, the winbindd implementation in use is that in the *samba* binary, from the Samba4 heritige. It shares the same protocol as the Samba3 [Winbind] in the *winbindd* binary, but does not implement the full protocol. It also implements a private IRPC based protocol for communication with other parts of the *samba* binary.
Currently in s3fs, the winbindd implementation in use is that in the '''samba''' binary, from the Samba4 heritage. It shares the same protocol as the Samba3 winbind in the '''winbindd''' binary, but does not implement the full protocol. It also implements a private IRPC based protocol for communication with other parts of the '''samba''' binary.


==Starting s3fs==
==Starting s3fs==
s3fs is started by the code in file_server/file_server.c, by writing a private fileserver.conf that includes a preamble of smb.conf options required for operation in this mode, and then it starts the smbd binary by fork() and exec().
s3fs is started by the code in file_server/file_server.c, by writing a private fileserver.conf that includes a preamble of smb.conf options required for operation in this mode, and then it starts the smbd binary by fork() and exec().


At provision time, run
At provision time, it is the default. To use the old ntvfs server, simply add the option:
<pre>--use-ntvfs</pre>
./provision --use-s3fs=yes

Or on later versions:

<pre>--with-ntvfs-fileserver</pre>

For instance run:
<pre> ./provision --domain=DEMOS3FS --realm=demos3fs.samba.corp --adminpass=P@ssw0rd</pre>

This creates an smb.conf with the equivalent of:
<pre>server services = -smb +s3fs
dcerpc endpoint servers = -winreg -srvsvc</pre>

You should add these options manually if you wish to move to s3fs on an existing production system.

Do note that NT ACLs set by s3fs will not be read by the ntvfs file server, if you switch back. This is because while both use the security.NTACL extended attribute, the s3fs module writes format version 3, not the original version 1 understood by the NTVFS server.


=Testing=
=Testing=


s3fs is tested in the *plugin_s4_dc* environment in *make test*. More tests need to be added
s3fs is tested in the '''plugin_s4_dc''' environment in '''make test'''. More tests need to be added


=Using it=
=Using it=
s3fs is the default. To use the previous default, the ntvfs server use --use-ntvfs at provision time or add these lines from the smb.conf
s3fs is not currently recommended for use, except where carefully configured by experienced developers or distributors in controlled environments. As we finish some more of this work, we hope to change this status soon.

<pre>
server services = +smb -s3fs
dcerpc endpoint servers = +winreg +srvsvc
</pre>

Latest revision as of 18:43, 12 May 2017

What is s3fs

s3fs is the name that was given to the development effort to make possible the agreed default file server configuration for Samba 4.0 as an AD Domain controller.

It was agreed at SambaXP 2010 that Samba 4.0 would release with the smbd file server in use by default, so that users upgrading from Samba 3.x DC environments to the 4.0 AD DC would still have access to all the features of the smbd file server environment that they had come to expect and rely on.

Why is this called s3fs?

The primary additional feature that s3fs gives our users is access to the SMB3 protocol developed in smbd. Additionally, this name communicates that this is about using the smbd file server from the Samba 3.x release stream.

For a number of years, two file servers were developed in parallel in the Samba project. The Samba 3.x releases produced the smbd file server, while the Samba 4.x alpha series had the ntvfs file server. The ntvfs server never gained more than early SMB2 support. Therefore, s3fs simply indicates that, integrated into the samba process used by the AD DC is the file server refined in the Samba 3.x release series and supporting SMB3.

It was hoped that this detail and name would not need to be exposed to users by the time that Samba 4.x was released, while we tried to refine the development and implementation details of this process.

How it is implemented

Because the smbd file server has a distinct history from the ntvfs file server that the Samba4 development project has used so far, it did not naturally use the same security subsystems and other resources common the the rest of the AD server. Indeed, at a time before the Franky effort was started, it was not even in the same GIT tree, and certainly not the same build tree.

The merged build

The merged or top level build is the Waf build system. Running ./configure and make at the top level of the source tree will build all of Samba, including the previously Samba3 components. As all duplicate symbol names have been merged, renamed or otherwise dealt with, the merged build has a large number of shared libraries that the whole project builds on, particularly the IDL generated file built using PIDL.

Common structures

Key structures describing the security, authentication and authorization state were changed to be in common. In particular, key structures like struct auth_session_info and struct security_token now describe the authentication and authorization state across the whole project.

Common subsystems

As part of the process of building s3fs, a large number of subsystems were made common, including in particular GENSEC. smbd now uses gensec for all 'blob-based' server-side authentication, in all protocols, which has made it possible to both move to a proper implementation of SPNEGO based on GSSAPI, and allowed the activated plugins to be easily switched for AD DC operation.

Named pipe forwarding

DCE/RPC servers in Samba have in the past been implemented in-process, simply being additional library code that implements the required service, within the forked smbd child process. However, for Samba 4.0 as an AD DC, there are technical reasons why the DCE/RPC servers need to be implemented in a long-lived manner. To communicate between the smbd process that handles file sharing and the DCE/RPC server, all the SMB named pipe operations are converted into operations on a unix domain socket. For AD services, other parts of the samba binary then listen on these private unix domain socket connections for SMB named pipe requests.

This also has the advantage that the AD services from the Samba4 heritage do not need to directly link into the smbd file server or the associated DCE/RPC stack. This approach was originally developed for the Franky effort for this reason, and remains key to how DCE/RPC is handled in Samba 4.0.

Other DCE/RPC services, such as spoolss are not forwarded, or are forwarded to another part of the smbd binary, allowing printing to function on an AD DC.

smb.conf configuration

The automatically generated fileserver.conf has lines such as:

rpc_server:samr = external

to indicate that these pipes are handled outside the smbd binary.

Plugins

With the merged build, it then becomes possible to load or link in plugins for key subsystems, to change the behaviour of the smbd file server, as required for consistency with the rest of the AD server.

auth_samba4

perhaps the most key plugin in auth_samba4. This plugin provides 2 very important hooks.

prepare_gensec

prepare_gensec returns a struct gensec_security that is pre-initialised to use the GENSEC modules that the rest of the AD server uses, including the authentication contexts etc required to authenticate against the AD directory in LDB. When the auth_samba4 module is not in use, the fallback is to the default authentication path, also expressed as a series of GENSEC modules.

make_auth4_context

make_auth4_context returns a struct auth4_context that similarly operates like the rest of the AD server, for SessionSetupAndX calls that do not use SPNEGO or NTLMSSP.

pdb_samba4

For operation of smbpasswd, pdbedit and net sam tools, and to assist in the migration of Samba 3.x DC environments to Samba 4.0 AD, a passdb module was written based on pdb_ads from Franky. This module differs in that it directly calls ldb and the associated modules, allowing offline operation when the samba server process is not running. This is particularly critical for the samba-tool domain samba3upgrade tool.

vfs_samba4

This module makes the ldb calls required to implement domain DFS referrals on the AD DC, and is loaded for operation on IPC$.

IDMAP

The idmap code in smbd has been modified to accept an ID mapping type of IDMAP_BOTH, representing both a uid and a gid. This will then allow a group (such as Domain Admins) to own a file as a uid, but also be a gid when expressed as group membership.

Winbind

Currently in s3fs, the winbindd implementation in use is that in the samba binary, from the Samba4 heritage. It shares the same protocol as the Samba3 winbind in the winbindd binary, but does not implement the full protocol. It also implements a private IRPC based protocol for communication with other parts of the samba binary.

Starting s3fs

s3fs is started by the code in file_server/file_server.c, by writing a private fileserver.conf that includes a preamble of smb.conf options required for operation in this mode, and then it starts the smbd binary by fork() and exec().

At provision time, it is the default. To use the old ntvfs server, simply add the option:

--use-ntvfs

Or on later versions:

--with-ntvfs-fileserver

For instance run:

 ./provision --domain=DEMOS3FS --realm=demos3fs.samba.corp --adminpass=P@ssw0rd

This creates an smb.conf with the equivalent of:

server services = -smb +s3fs
dcerpc endpoint servers = -winreg -srvsvc

You should add these options manually if you wish to move to s3fs on an existing production system.

Do note that NT ACLs set by s3fs will not be read by the ntvfs file server, if you switch back. This is because while both use the security.NTACL extended attribute, the s3fs module writes format version 3, not the original version 1 understood by the NTVFS server.

Testing

s3fs is tested in the plugin_s4_dc environment in make test. More tests need to be added

Using it

s3fs is the default. To use the previous default, the ntvfs server use --use-ntvfs at provision time or add these lines from the smb.conf

server services = +smb -s3fs
dcerpc endpoint servers = +winreg +srvsvc