Difference between revisions of "SoC/Ideas"

From SambaWiki
(Add GVFS idea.)
(last moment Samba 4 tasks)
Line 42: Line 42:
   
 
==Samba 4==
 
==Samba 4==
  +
  +
===LDAP Backend===
  +
These associated projects could be combined into one proposal, or taken on individually by others
  +
====Merge provision-backend and provision====
  +
  +
The current two-step process of provision-backend and a separate
  +
provision seems to cause a lot of challenges for folks. They often get
  +
the required command line arguments wrong, and have to manually start
  +
slapd before we can provision the data.
  +
  +
Similarly, when we provision against Fedora DS, it would prefer to start
  +
the server right away.
  +
  +
An improved provision script would take a path to the slapd binary
  +
(possibly detected at configure time), and incorporate the code in
  +
master/selftest/target/Samba4.pm to generate the modules.conf for the
  +
slapd.conf
  +
  +
Then, the python script should start slapd, and watch it for improper
  +
termination. If it starts (using the command line already suggested,
  +
but only listening on ldapi) and stays started, we should provision it
  +
normally, then shut it down.
  +
  +
Then, knowing everything worked, we can suggest how the user can start
  +
smbd and slapd as required. (the output of this could be put in a file
  +
that 'make test' can then parse, to ensure it verifies this value).
  +
  +
As a bonus, if it remains possible to test the current provision-backend
  +
code, when a valid OpenLDAP installation is not present, all the better
  +
(help prevent bitrot).
  +
  +
*Difficulty: Medium
  +
*Language(s): python
  +
*Possible mentors: Andrew Bartlett
  +
  +
====Test OpenLDAP in a normal 'make test'====
  +
  +
Detect that OpenLDAP 2.4.15 is present on the system, and it's location.
  +
If so, when 'make test' is run, then 'make quicktest' is additionally
  +
run against the LDAP backend.
  +
  +
This should help reduce bitrot in the OpenLDAP backend because it fails
  +
to be tested.
  +
  +
Currently testing against the OpenLDAP backend requires that you run:
  +
  +
TEST_LDAP=yes OPENLDAP_ROOT=/usr/local make test
  +
  +
Most developers don't do this regularly (either because they don't have
  +
the right OpenLDAP, or they don't know about it, or they don't think
  +
to).
  +
  +
*Difficulty: Medium
  +
*Language(s): python
  +
*Possible mentors: Andrew Bartlett
  +
  +
====Restore the Fedora DS backend====
  +
  +
Currently the Fedora DS backend won't even start. Small details have
  +
changed in both Fedora DS and Samba4, and this code has bit-rotted. The
  +
task would be to make it pass 'make test' with as few failures as
  +
possible (some are inevitable, as it has a different feature set to
  +
OpenLDAP).
  +
  +
*Difficulty: Medium
  +
*Language(s): python, C
  +
*Possible mentors: Andrew Bartlett
  +
  +
====Add correct indexes to OpenLDAP and Fedora DS====
  +
  +
Currently the performance of Samba4 against the OpenLDAP backend isn't
  +
very good.
  +
  +
Part of this might be because we don't set up the correct indexes. I'm
  +
wondering if someone might like to modify Samba4's provision-backend
  +
system (see scripting/python/samba/provision.py:provision_backend()) to
  +
generate indexes in the same way it generates refint and memberof
  +
configuration.
  +
  +
This would replace the hard-coded indexes in setup/slapd.conf
  +
  +
We already generate the index for local ldb in
  +
source/dsdb/schema/schema_set.c:dsdb_schema_set_attributes(), but this
  +
is via an abstraction layer that is probably a bit painful to use in
  +
provision.py.
  +
  +
The flags are in the attribute searchFlags and the values are defined in
  +
source4/dsdb/common/flags.h (not sure if these are easily brought into
  +
python).
  +
  +
A bonus task would be to also enforce attribute uniqueness using the
  +
uniqneness overlay and in LDB.
  +
  +
*Difficulty: Medium
  +
*Language(s): python, C for LDB unique indexes.
  +
*Possible mentors: Andrew Bartlett
  +
  +
===Syntax correctness===
  +
The Samba4 LDB layer uses the Microsoft AD schema to determine what data
  +
types apply to each attribute.
  +
  +
The schema contains references to oMsyntax, oMobjectClass and
  +
attributeSyntax, that we then correlate with our internal data type
  +
tables.
  +
  +
The table for this is in dsdb/schema/schema_syntax.c (and then this
  +
referrs to parts of ldb and elsewhere for some syntaxes).
  +
  +
The problem is, many of these data types are not handled correctly in
  +
replication (anything with a DATA_BLOB conversion to and from DRS is
  +
suspect), and even if they are, we do not apply strict tests for
  +
conformance of the data to to the data type when adding it to the
  +
database (such as via LDB or LDAP).
  +
  +
The task for the student or otherwise interested onlooker would be to
  +
investigate the comparison and conversion rules for each type of
  +
attribute, and to write tests to prove they are correct. This should
  +
include for the DRSUAPI replication of the data. Tests that do a DRS
  +
replication against windows, and then prove we still present the correct
  +
values in LDAP are also required.
  +
  +
See the wspp-schema in abartlet's GIT area for our current work in
  +
progress (making changes to how the schema is loaded and applied to
  +
LDB), which will be a clearer basis for this work. It also indicates
  +
how to add new comparison rules. (or master-devel once it's merged)
  +
  +
Smaller parts of this (and it would need to be done in parts anyway)
  +
could be done by anyone interested in the correctness of Samba4's LDAP
  +
server.
  +
  +
See also bugs in bugzilla such as https://bugzilla.samba.org/show_bug.cgi?id=6136 regarding incorrect handling of 32 bit
  +
integers in our LDAP server.
  +
  +
*Difficulty: Medium
  +
*Language(s): C, python for tests
  +
*Possible mentors: Andrew Bartlett
  +
  +
===Admin Utilities===
  +
We still need a few more Samba4-specific administration utilities to make Samba4 useful in real life. A Summer of Code student would be expected to do a number of these (identifying new needs from deployments), but here is a suggestion for one:
  +
====Password Properties Script====
  +
We currently have utilities in source4/setup such as newuser,
  +
setpassword and setexpiry.
  +
  +
However, we need a utility to manipulate the password properties, such
  +
as minPwdAge, maxPwdAge, minPwdLength and pwdProperties. These are
  +
found in the basedn of the domain (ie DC=samba,DC=org).
  +
  +
pwdProperties in particular is a bitmask, defined as:
  +
  +
/* password properties flags */
  +
typedef [public,bitmap32bit] bitmap {
  +
DOMAIN_PASSWORD_COMPLEX = 0x00000001,
  +
DOMAIN_PASSWORD_NO_ANON_CHANGE = 0x00000002,
  +
DOMAIN_PASSWORD_NO_CLEAR_CHANGE = 0x00000004,
  +
DOMAIN_PASSWORD_LOCKOUT_ADMINS = 0x00000008,
  +
DOMAIN_PASSWORD_STORE_CLEARTEXT = 0x00000010,
  +
DOMAIN_REFUSE_PASSWORD_CHANGE = 0x00000020
  +
} samr_PasswordProperties;
  +
  +
I need a new script to make it easier for administrators to set their
  +
domain to not enforce complex passwords (in particular), and to set the
  +
other flags.
  +
  +
*Difficulty: Easy
  +
*Language(s): Python
  +
*Possible mentors: Andrew Bartlett
   
 
===GQ replacement using GTK and LDB===
 
===GQ replacement using GTK and LDB===

Revision as of 03:13, 3 April 2009

=Google Summer of Code: Suggested Project ideas=

The following are the Samba project ideas for Summer of Code. Of course you are free to come up with ideas not listed here. Please discuss the your planned project by either joining us on irc://irc.freenode.net/#samba-technical or by sending email to samba-technical@samba.org

Most of our projects will require C programming skills, but the Samba4 section has a couple of Python projects.

Samba 3

Add remote (RPC) support for Samba configuration

Since recently, Samba has a registry based configuration backend: Configuration data is stored inside the registry key HKEY_LOCAL_MACHINE\Software\Samba\smbconf. Access to this configuration is available through a module that makes use of the "reg_api" interface which is similar to the winreg API. This project would at first develop a common api for registry access local through reg_api and remote through the winreg rpc client code. This new module could be abstraced from the code of the utility "net rpc registry". With this new module, the registry configuration code could be enhanced to allow for remote configuration.

  • Difficulty: Medium
  • Language(s): C
  • Possible Mentors: Michael Adam

Backport EndPointMapper and support for ncacn_ip_tcp

Current Samba 3.0 releases only support MS-RPC over SMB named pipes. The SAMBA_4_0 code base has support RPC directory over TCP which could be back ported in some degree to the SAMBA_3_0 tree. A pre-requisite task may be to backport the events framework first

Remove use of unexpected.tdb

Current Samba 3 contains the unexpected.tdb as a mechanism to pass UDP packets coming in on ports 137 and 138 to interested client applications. This mechanism needs to be replaced by a unix domain socket on which nmbd offers a lightweight RPC interface to send and receive nmb name service packets and send and receive mailslot requests.

  • Difficulty: Hard
  • Language(s): C
  • Possible mentors: Volker Lendecke

Samba 4

LDAP Backend

These associated projects could be combined into one proposal, or taken on individually by others

Merge provision-backend and provision

The current two-step process of provision-backend and a separate provision seems to cause a lot of challenges for folks. They often get the required command line arguments wrong, and have to manually start slapd before we can provision the data.

Similarly, when we provision against Fedora DS, it would prefer to start the server right away.

An improved provision script would take a path to the slapd binary (possibly detected at configure time), and incorporate the code in master/selftest/target/Samba4.pm to generate the modules.conf for the slapd.conf

Then, the python script should start slapd, and watch it for improper termination. If it starts (using the command line already suggested, but only listening on ldapi) and stays started, we should provision it normally, then shut it down.

Then, knowing everything worked, we can suggest how the user can start smbd and slapd as required. (the output of this could be put in a file that 'make test' can then parse, to ensure it verifies this value).

As a bonus, if it remains possible to test the current provision-backend code, when a valid OpenLDAP installation is not present, all the better (help prevent bitrot).

  • Difficulty: Medium
  • Language(s): python
  • Possible mentors: Andrew Bartlett

Test OpenLDAP in a normal 'make test'

Detect that OpenLDAP 2.4.15 is present on the system, and it's location. If so, when 'make test' is run, then 'make quicktest' is additionally run against the LDAP backend.

This should help reduce bitrot in the OpenLDAP backend because it fails to be tested.

Currently testing against the OpenLDAP backend requires that you run:

TEST_LDAP=yes OPENLDAP_ROOT=/usr/local make test

Most developers don't do this regularly (either because they don't have the right OpenLDAP, or they don't know about it, or they don't think to).

  • Difficulty: Medium
  • Language(s): python
  • Possible mentors: Andrew Bartlett

Restore the Fedora DS backend

Currently the Fedora DS backend won't even start. Small details have changed in both Fedora DS and Samba4, and this code has bit-rotted. The task would be to make it pass 'make test' with as few failures as possible (some are inevitable, as it has a different feature set to OpenLDAP).

  • Difficulty: Medium
  • Language(s): python, C
  • Possible mentors: Andrew Bartlett

Add correct indexes to OpenLDAP and Fedora DS

Currently the performance of Samba4 against the OpenLDAP backend isn't very good.

Part of this might be because we don't set up the correct indexes. I'm wondering if someone might like to modify Samba4's provision-backend system (see scripting/python/samba/provision.py:provision_backend()) to generate indexes in the same way it generates refint and memberof configuration.

This would replace the hard-coded indexes in setup/slapd.conf

We already generate the index for local ldb in source/dsdb/schema/schema_set.c:dsdb_schema_set_attributes(), but this is via an abstraction layer that is probably a bit painful to use in provision.py.

The flags are in the attribute searchFlags and the values are defined in source4/dsdb/common/flags.h (not sure if these are easily brought into python).

A bonus task would be to also enforce attribute uniqueness using the uniqneness overlay and in LDB.

  • Difficulty: Medium
  • Language(s): python, C for LDB unique indexes.
  • Possible mentors: Andrew Bartlett

Syntax correctness

The Samba4 LDB layer uses the Microsoft AD schema to determine what data types apply to each attribute.

The schema contains references to oMsyntax, oMobjectClass and attributeSyntax, that we then correlate with our internal data type tables.

The table for this is in dsdb/schema/schema_syntax.c (and then this referrs to parts of ldb and elsewhere for some syntaxes).

The problem is, many of these data types are not handled correctly in replication (anything with a DATA_BLOB conversion to and from DRS is suspect), and even if they are, we do not apply strict tests for conformance of the data to to the data type when adding it to the database (such as via LDB or LDAP).

The task for the student or otherwise interested onlooker would be to investigate the comparison and conversion rules for each type of attribute, and to write tests to prove they are correct. This should include for the DRSUAPI replication of the data. Tests that do a DRS replication against windows, and then prove we still present the correct values in LDAP are also required.

See the wspp-schema in abartlet's GIT area for our current work in progress (making changes to how the schema is loaded and applied to LDB), which will be a clearer basis for this work. It also indicates how to add new comparison rules. (or master-devel once it's merged)

Smaller parts of this (and it would need to be done in parts anyway) could be done by anyone interested in the correctness of Samba4's LDAP server.

See also bugs in bugzilla such as https://bugzilla.samba.org/show_bug.cgi?id=6136 regarding incorrect handling of 32 bit integers in our LDAP server.

  • Difficulty: Medium
  • Language(s): C, python for tests
  • Possible mentors: Andrew Bartlett

Admin Utilities

We still need a few more Samba4-specific administration utilities to make Samba4 useful in real life. A Summer of Code student would be expected to do a number of these (identifying new needs from deployments), but here is a suggestion for one:

Password Properties Script

We currently have utilities in source4/setup such as newuser, setpassword and setexpiry.

However, we need a utility to manipulate the password properties, such as minPwdAge, maxPwdAge, minPwdLength and pwdProperties. These are found in the basedn of the domain (ie DC=samba,DC=org).

pwdProperties in particular is a bitmask, defined as:

       /* password properties flags */
       typedef [public,bitmap32bit] bitmap {
               DOMAIN_PASSWORD_COMPLEX         = 0x00000001,
               DOMAIN_PASSWORD_NO_ANON_CHANGE  = 0x00000002,
               DOMAIN_PASSWORD_NO_CLEAR_CHANGE = 0x00000004,
               DOMAIN_PASSWORD_LOCKOUT_ADMINS  = 0x00000008,
               DOMAIN_PASSWORD_STORE_CLEARTEXT = 0x00000010,
               DOMAIN_REFUSE_PASSWORD_CHANGE   = 0x00000020
       } samr_PasswordProperties;

I need a new script to make it easier for administrators to set their domain to not enforce complex passwords (in particular), and to set the other flags.

  • Difficulty: Easy
  • Language(s): Python
  • Possible mentors: Andrew Bartlett

GQ replacement using GTK and LDB

GQ is a widely used LDAP query tool. Many LDAP administrators would benefit if a similar tool were constructed, in particular with similar schema knowledge. There is a LDB-based clone of it in Samba-Gtk. We would like to see this tool extended and e.g. having support for adding and editing entries.

Extension of the GTK+ frontends

There are a couple of GTK+ frontends for Samba4 (see SambaGtk). These are very limited at the moment but you could work on expanding them and further integrating them with GNOME. Language: C or Python

  • Difficulty: Easy
  • Language(s): Python, perhaps C
  • Possible mentors: JelmerVernooij

SWAT implementation in Python

SWAT, the Samba Web Administration Tool is basically nonexistant in Samba4 at the moment. We would like to see it implemented in Python for Samba 4, with a focus on usability.

  • Difficulty: Easy
  • Language(s): Python, JavaScript (Ajax)
  • Possible mentors: JelmerVernooij

SMB and SMB2 backend for GVFS based on libcliraw

GVFS is the GNOME File System abstraction layer, and used (among other things) in Nautilus. Its current SMB backend does not support the SMB2 protocol and is restricted to the (limited) API of libsmbclient.

A useful project would be to add a new backend for GVFS based on Samba 4's SMB client libraries. This would add support for SMB2, and would cut out the mapping to POSIX in between.

  • Difficulty: Medium
  • Language(s): C
  • Possible mentors: JelmerVernooij, ?

Windows Search Protocol WSP client library and torture tests

The Windows Search Protocol WSP is used to implement remote full filesystem indexing (indexed search) between windows machines. We would like to support this functionality in Samba, interfacing with existing indexing tools on Unix systems (such as beagle).

This is a new protocol based on SMB named pipes \pipe\ci_skads or \pipe\MSFTEWDS. See http://msdn2.microsoft.com/en-us/library/cc216195.aspx.

The student should write a (un)marshalling library to push and pull PDUs and an async client library on top of the samba4 raw smb client library.

The student should write sub tests for smbtorture which should demostrate how the protocol works against a windows server.

The student doesn't have to implement the samba4 server code.

  • Difficulty: Medium
  • Language(s): C, (Python?)
  • Possible Mentors: Tridge

CIFS POSIX extensions in Samba4

Samba4 does not implement the CIFS POSIX extensions at this stage.

A testsuite needs to be written, to prove consistent behavior, and the Samba3 compatible server implemented.

  • Difficulty: Medium
  • Language(s): C
  • Possible mentors: Tridge?

Re-implement smbclient in python

Samba4 now has an embedded python interpreter, which is used for both web configuration and for command line tools. The interpreter has access to the extensive internal C library of Samba. We would like smbclient to be rewritten in python, making it much more easily extendable by administrators.

  • Difficulty: Medium
  • Language(s): Python, C
  • Possible mentors: JelmerVernooij

Browsing support in Samba 4

Samba 4 still needs support for mailslots in general and in particular for the BROWSE mailslot. Should come with tests. Documentation of the BRWOSER protocol is available here: http://msdn.microsoft.com/en-us/library/cc201609(PROT.10).aspx http://ubiqx.org/cifs/Browsing.html

  • Difficulty: Hard
  • Language(s): C
  • Possible mentors: JelmerVernooij, Stefan Metzmacher

Miscellaneous

Linux Kernel CIFS client improvements

Interested students should contact Steve French and discuss possible improvements to the Linux Kernel CIFS VFS client. Here are some ideas to get you started:

  • improved async/vectored i/o support (improves performance)
  • Create a GUI for creating and managing Linux cifs mounts, and more easily configuring the many complex cifs mount options
  • Create GUI for displaying /proc/fs/cifs statistics and and mount/session status
  • Create a common uid mapping mechanism for Linux nfs and cifs vfs clients, that can be configured to remap, on a particular mount on the client, the uids returned by the server to uids which would be valid on the client (or to a default if such uid does not exist). This is helpful when the server even if it supports the CIFS Unix Extensions has different uids and gids than the client
  • Finish the Linux cifs vfs change notification support by completing the D_NOTIFY fcntl mapping to the FindNotify mechanism supported by the CIFS protocol
  • prototype SMB2 client
  • integration of cifs client with Dave Howell's fscache/cachefs (for offline caching of files on the client and to allow disconnected operation)
  • cifs->Samba automated test facility (build verification) similar to what we can now do with the Samba server and tools in the Samba build farm
  • Support for alternate transport protocols (other than TCP sockets)
  • Support for features (such as directory delegations) which NFS version 4.1 has but which current CIFS even with the most current CIFS->Samba protocol extensions (CIFS Unix Extensions) do not have.
  • Support for retrieving snapshots, encrypted files, or compressed files from Windows via the CIFS VFS


  • Difficulty: Varies
  • Language(s): C
  • Possible mentors: Steve French