Difference between revisions of "SoC/Ideas"

(SMB2 protocol improvements)
(Print System Asynchronous Remote Protocol Wireshark Dissectors)
 
(73 intermediate revisions by 13 users not shown)
Line 4: Line 4:
 
Of course you are free to come up with ideas not listed here.
 
Of course you are free to come up with ideas not listed here.
 
Please discuss the your planned project by either joining us on irc://irc.freenode.net/#samba-technical or  
 
Please discuss the your planned project by either joining us on irc://irc.freenode.net/#samba-technical or  
by sending email to samba-technical@samba.org
+
by sending email to [https://lists.samba.org/mailman/listinfo/samba-technical samba-technical@lists.samba.org]
  
Most of our projects will require C programming skills, but the Samba4 section has a couple of Python projects.
+
==Samba==
  
==Samba 3==
+
Some additional possible GSoC topics can be found in Bugzilla in the form of bugs which are marked as "Feature request": [https://bugzilla.samba.org/buglist.cgi?query_format=advanced&short_desc=Feature%20request&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&short_desc_type=allwordssubstr&product=Samba%204.0 here]. Questions regarding complexity and requirements should be directed to the technical mailing list.
 +
 
 +
<!-- Commented out possibly stale proposals
  
There are currently no project proposals for this area.
+
===Browsing support in Samba 4===
 +
Samba 4 still needs support for mailslots in general and in particular for the BROWSE mailslot. Should come with tests. Documentation of the BROWSER protocol is available here:
 +
http://msdn.microsoft.com/en-us/library/cc201609(PROT.10).aspx
 +
http://ubiqx.org/cifs/Browsing.html
  
==Samba 4==
+
*Difficulty: Hard
 +
*Language(s): C
 +
*Possible mentors: Stefan Metzmacher
  
Some additional possible GSoC topics can be found in Bugzilla in the form of bugs which are marked as "Feature request": [https://bugzilla.samba.org/buglist.cgi?query_format=advanced&short_desc=Feature%20request&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&short_desc_type=allwordssubstr&product=Samba%204.0 here]. Questions regarding complexity and requirements should be directed to the technical mailing list.
+
===Implement login / logout related counter update===
 +
For the moment the attributes related to login and logout are not
 +
updated by Samba4.
 +
The goal of this project is to understand in which case windows update
 +
the counters (ie. most probably during interactive logon but also maybe
 +
with some netlogon calls ?) and to implement counter and timestamp
 +
update is Samba code so that this information can be available.
 +
This project of course includes the development of unit tests.
 +
 
 +
*Difficulty: Easy
 +
*Language(s): C
 +
*Possible mentors: Andrew Bartlett
 +
 
 +
===Improve regedit===
 +
 +
Last year someone has started to write a ncurses based registry editor. The editor could be improved, like put some effort in a better look and feel and adding dcerpc winreg support to remotely connect to the registry.
  
===Admin Utilities===
+
See https://git.samba.org/?p=asn/samba.git;a=shortlog;h=refs/heads/regedit
  
We still need a few more Samba4-specific administration utilities to make Samba4 useful in real life. A Summer of Code student would be expected to do a number of these (identifying new needs from deployments), but here are some suggestions:
+
* Difficulty: Medium
 +
* Language(s): C
 +
* Possible mentors: Andreas Schneider, Michael Adam
  
===Setup / provision GUI for Samba (and OpenChange)===
 
  
Setting up Samba and OpenChange currently requires running quite a lot of command line utilities. It should be easy for even a junior system administrator to set up Samba as a Domain Controller, RODC and general server.
+
===Utilize libsmbclient server-side copy support in file managers===
  
The windows equivalent tool is dcpromo.exe, and while you may not want/need to copy it directly, it does provide a reasonable basis for establishing what this tool should be able to provide (in terms of outcomes).
+
With libsmbclient now supporting server-side copy requests via [https://git.samba.org/?p=samba.git;a=commit;h=f73bcf4934be89f83e86459bc695b7d28348565c cli_splice()], file managers making use of libsmbclient can be changed to utilize server-side copy support for greatly improved remote copy performance. Potential file manager targets include [https://bugzilla.gnome.org/show_bug.cgi?id=771022 GNOME Files/Nautilus (gvfs_smb)], Dolphin (kio_smb) and Kodi's File Manager.
  
Some initial work (using PyQt4) is available at http://gitorious.org/samba-openchange-gui-tools/samba-openchange-gui-setup but working with that is not essential to this task.
+
*Difficulty: Easy, Medium
 +
*Language(s): C, C++
 +
*Possible mentors: David Disseldorp
  
*Difficulty: Easy
 
*Language(s): Python
 
*Possible mentors: Brad Hards, probably others.
 
  
 
===Windows Search Protocol WSP client library and torture tests===
 
===Windows Search Protocol WSP client library and torture tests===
  
The Windows Search Protocol WSP is used to implement remote full filesystem
+
The Windows Search Protocol WSP is used to implement remote full filesystem indexing (indexed search) between Windows machines. We would like to support this functionality in Samba, interfacing with existing indexing tools on Unix systems (such as GNOME Tracker).
indexing (indexed search) between windows machines. We would like to
 
support this functionality in Samba, interfacing with existing
 
indexing tools on Unix systems (such as beagle).
 
  
This is a new protocol based on SMB named pipes
+
This is a DCE/RPC protocol. See http://msdn.microsoft.com/en-us/library/cc251767.aspx .
\pipe\ci_skads or \pipe\MSFTEWDS.
 
See http://msdn2.microsoft.com/en-us/library/cc216195.aspx.
 
  
The student should write a (un)marshalling library
+
The student should write a (un)marshalling library to push and pull PDUs and an asynchronous client library on top of the Samba raw smb client library.
to push and pull PDUs and an async client library
 
on top of the samba4 raw smb client library.
 
  
The student should write sub tests for smbtorture
+
The student should write sub-tests for smbtorture which should demonstrate how the protocol works against a Windows server.
which should demostrate how the protocol works against
+
The student doesn't have to implement the Samba server code.
a windows server.
+
Noel Power from SUSE has done some basic server implementation, he should be able to give guidance
  
The student doesn't have to implement the samba4 server code.  
+
*Difficulty: Medium, Hard
 +
*Language(s): C, (Python)
 +
*Possible Mentors: Noel Power
 +
 
 +
-->
 +
=== Print System Asynchronous Remote Protocol Wireshark Dissectors ===
 +
 
 +
The Print System Asynchronous Remote Protocol ([https://msdn.microsoft.com/en-us/library/cc238080.aspx MS-PAR]) is a replacement for the synchronous Print System Remote Protocol (MS-RPRN). MS-PAR inherits many message and buffer formats from the old protocol, but allows for asynchronous submission and notification of print jobs. Further details of the protocol can be found in Günther and Andreas' [http://archive.sambaxp.org/fileadmin/user_upload/SambaXP2013-DATA/thu/track2/Guenther_Deschner_Andreas_Schneider-Printing_Samba_4.pdf SambaXP presentation].
 +
 
 +
The student should write Wireshark dissectors for MS-PAR.
 +
The student should improve existing smbtorture tests, which demonstrate how the protocol works against a Windows server.
 +
 
 +
*Difficulty: Medium
 +
*Language(s): C
 +
*Possible Mentors: Andreas Schneider (supported by Günther Deschner)
 +
 
 +
===dbwrap back-end for Ceph RADOS key-value storage===
 +
 
 +
Ceph offers a highly scalable and fault-tolerant storage system. Samba is already capable of sharing data located on the [https://ceph.com/ceph-storage/file-system/ Ceph Filesystem], however scale-out sharing (the same data exposed by multiple Samba nodes) currently requires the use of [https://ctdb.samba.org/ CTDB] for consistent and coherent state across Samba cluster nodes. In such a setup CTDB provides a clustered database with persistent key-value data storage and locking. Database usage is abstracted out via a generic ''dbwrap'' interface.<br>
 +
 
 +
Ceph's librados library provides an API for the storage and retrieval of arbitrary key-value data via the ''omap'' functions. A watch/notify protocol is also provided as a mechanism for synchronising client state (locking). Key-value data stored in the RADOS back-end inherits the same redundancy features as regular objects, making it a potentially good candidate as a replacement for CTDB in scale-out Samba clusters.
 +
 
 +
This task involves the implementation and testing of a new ''dbwrap'' back-end that uses librados for the storage, retrieval and locking of Samba key-value state. Ideally, the candidate would also allow time for benchmarking.
 +
 
 +
*Difficulty: Medium
 +
*Language(s): C
 +
*Possible Mentors: David Disseldorp
 +
 
 +
 
 +
=== Samba binary size reduction ===
 +
 
 +
Samba has grown to quite a bloated beast. This task will focus on some areas where the bloat can easily be reduced, e.g.
 +
* Removal of unused autogenerated librpc code
 +
** RPC client and server code when only serialization functions are used
 +
** optional struct print routines (pidl noprint?)
 +
** Some knowledge of perl would help here (for pidl)
 +
* Add new build options to compile Samba without certain functionality
 +
** Undesired DCE/RPC services
 +
** Printing support
 +
** Legacy SMB/CIFS support (stretch goal)
 +
 
 +
 
 +
*Difficulty: easy
 +
*Language(s): Some knowledge of C and python would be helpful
 +
*Possible Mentors: David Disseldorp (supported by Andrew Bartlett)
 +
 
 +
=== Fuzz Samba using [http://lcamtuf.coredump.cx/afl/ American Fuzzy Lop] ===
 +
 
 +
Try to find bugs in the server or client components of Samba using [http://lcamtuf.coredump.cx/afl/ AFL]. This is not the usual AFL scenario where you can fuzz a file format parser. It needs to be a bit smart about it, as the way the SMB protocol works, you need to do a series of steps before reaching arbitrary SMB commands (protocol negotiation, session setup, tree connect, ...). Similarly some checks like packet signing should be worked around to reach the best results from AFL.
 +
 
 +
Samba code will need to be modified in hackish ways to make this work. I don't expect to be able to merge it back but if some of the modification required could be cleaned up and integrated in would be a bonus.
 +
 
 +
* Difficulty: medium
 +
* Language(s): Some knowledge of C
 +
* Possible Mentors: Aurélien Aptel
 +
 
 +
===Samba AD DC as the ideal POSIX Directory===
 +
 
 +
Samba is a great Active Directory Domain Controller, but it is not an ideal directory server for a large, passionate and important user base:  Sites with Samba SMB servers, but also general purpose Linux servers.  A smaller subset of these sites also have Linux desktops.  These sites may also have Windows servers, but they like the Windows desktops, are not the focus.
 +
 
 +
These sites often used Samba + OpenLDAP, and are finding the move to Samba's AD DC a bit difficult, because schema extension is hard, some things are not done automatically (like uidNumber allocation), and in general the focus has been around matching Windows not listening to the needs of this part of our user base.
 +
 
 +
Specific research should be done into what FreeIPA does well in targeting this user segment, and what customisations advanced users of OpenLDAP apply.
 +
 
 +
This project would be to propose a number of specific improvements, and to add both tests and an implementation of these improvements to Samba.
  
 
*Difficulty: Hard
 
*Difficulty: Hard
*Language(s): C, (Python?)
+
*Languages(s): C, Python
*Possible Mentors: Tridge
+
*Possible Mentors: Andrew Bartlett
 +
 
 +
 
 +
===GitLab CI of Samba for non-linux platforms (FreeBSD in particular)===
 +
 
 +
Samba uses GitLab CI to improve the quality of our patches.  Efforts are currently underway to extend the docker container used from just Ubuntu 14.04 to later versions and other distributions.
 +
 
 +
However, we occasionally have issues ensuring Samba still builds and operates on FreeBSD and other non-linux platforms.
 +
 
 +
The idea would be to have a docker image and .gitlab-ci.yml code to support it that runs FreeBSD and then builds and runs Samba's testsuite inside that FreeBSD nested VM, while still outputting the results to the normal gitlab-ci.
 +
 
 +
This differs from just running GitLab CI runners on FreeBSD as we need auto-scale, destroy the host and guest at the end of the test and run on Linux docker (such as the free GitLab,com CI runners).
 +
 
 +
As a stretch goal, being able to run some tests against a specific linux kernel and a raw ext4 filesystem (rather than unionfs) via qemu rather than depending on the docker host configuraiton.
 +
 
 +
*Difficulty: Medium
 +
*Language(s): Python, shell, YAML
 +
*Possible Mentors: Andrew Bartlett
 +
 
 +
===Integrate Samba AD DC deployment and management with Cockpit===
 +
 
 +
A prototype at https://github.com/abbra/cockpit-app-samba-ad shows how we can integrate Samba AD deployment with Cockpit Linux management console. A goal of this task is to move forward with this prototype to produce a fully working Samba AD management tool for common operations supported by 'samba-tool' command line utility.
 +
 
 +
*Difficulty: Medium
 +
*Language(s): Python, JavaScript (React), CSS, HTML
 +
*Possible Mentors: Alexander Bokovoy
 +
 
 +
 
 +
<!-- Commented out possibly stale proposals
 +
 
 +
===Make libsmbclient thread-safe for Gnome VFS===
  
===Browsing support in Samba 4===
+
libsmbclient is currently not thread-safe, even when different threads use different libsmbclient contexts. This has a big impact on use by the Gnome VFS code. The easiest fix is to add a "Big Lock" around all elements of libsmbclient that are not currently thread safe. This is not fine-grained threading support, but would add mutexes to creating and any manipulation of contexts. Locks also need to be added around any calls into the parameter reading/writing subsystems, and many other places within libsmbclient. Success would be to report safe hellgrind usage on test code using multiple libsmbclient contexts to access common files from a single server/share.
Samba 4 still needs support for mailslots in general and in particular for the BROWSE mailslot. Should come with tests. Documentation of the BROWSER protocol is available here:
 
http://msdn.microsoft.com/en-us/library/cc201609(PROT.10).aspx
 
http://ubiqx.org/cifs/Browsing.html
 
  
*Difficulty: Hard
+
*Difficulty: Medium, Hard
 
*Language(s): C
 
*Language(s): C
*Possible mentors: [[JelmerVernooij]], Stefan Metzmacher
+
*Possible mentors: Jeremy Allison
 +
-->
  
===Make samba 4 DC Group Policies (GPO) aware===
+
==Linux Kernel SMB Client Improvements==
Currently Samba 4 DC is able to serve GPOs to clients and they are mostly able to act according to the content of those GPOs.
 
But even if the GPO concerns AD DCs, Samba 4 ignore them even if some parameters are meaningful in a Samba 4 context (ie. password length, password life ...).https://wiki.samba.org/index.php?title=SoC/Ideas&action=edit&section=9
 
In order to work around this limitation, there is currently a couple of scripts that allow to set them but it's a suboptimal experience.
 
  
The goal of this project is to make Samba 4 periodically check if there is a GPO for it, check if any parameter of this GPO are meaningful for Samba (as a counter example a GPO which defines the background color of the Desktop on DC is not meaningful for Samba 4) and if so to alter parameters accordingly.
+
The Linux Kernel has a module called cifs.ko which is independent from Samba (it doesn't share code) that allows users to mount remote shares. It supports multiple dialects of SMB (1, 2, 3). The protocol dialects are now officially documented by Microsoft (See [https://msdn.microsoft.com/en-us/library/cc246231.aspx MS-SMB], [https://msdn.microsoft.com/en-us/library/cc246482.aspx MS-SMB2]) so students shouldn't have to worry about reverse engineering to understand them. The [https://www.wireshark.org/ Wireshark] open source network sniffer&dissector is a very good learning tool as well.
More details can be found in [http://lists.samba.org/archive/samba-technical/2010-April/070296.html Matthieu's samba-technical email].
 
  
*Difficulty: Easy, Medium
 
*Language(s): C, Python
 
*Possible mentors: Matthieu Patou
 
  
===Implement login / logout related counter update===
+
Interested students should contact Steve French or the [mailto:linux-cifs@vger.kernel.org linux-cifs mailing list] to discuss possible improvements to the Linux Kernel CIFS VFS client.
For the moment the attributes related to login and logout are not
+
 
updated by Samba4.
+
=== Add machine-readable debug & stats /proc file ===
The goal of this project is to understand in which case windows update
+
* We currently output debug and statistic information under /proc/fs/cifs/ (DebugData, Stats, ...). We need to stop outputing free format text that breaks all parsers out there everytime we add things to it. Clean up the cifsdebug.c file (it is kind of messy). Possibly generate a hierarchy of /proc files (e.g. a dir per tcp connection, subdirs for session, files for tcons) instead of dumping everything in one file.
the counters (ie. most probably during interactive logon but also maybe
+
* Make a nice visualizer/dashboard thing to get an overview. This could be console or GUI or...
with some netlogon calls ?) and to implement counter and timestamp
+
* Language: C for the kernel stuff, Userspace can be C, C++, Python.
update is Samba code so that this information can be available.
+
* Difficulty: Low
This project of course includes the development of unit tests.
+
 
 +
=== Add additional ftrace (trace-cmd) tracepoints and better GUI ===
 +
* Add more dynamic tracepoints to cifs.ko for commonly needed user scenarios, and add GUI (and/or CLI) tool to make it easier to enable/disable cifs.ko dynamic trace points (See /sys/kernel/debug/tracing/events/cifs/ for the pseudo-files that are currently configured manually for tracing or via trace-cmd) Make a nice native/console/web UI for it.
 +
* Language: C (any kernel changes) and userspace C, C++ or Python
 +
* Difficulty: Low
 +
 
 +
=== Add performance analysis cli tools ===
 +
* Add more perf tools for SMB3 client (similar to iostat or nfsstat) that leverage (and possibly extend what is captured) in /proc/fs/cifs/Stats but make it easier to analyze performance of a cifs mount
 +
* Language: C, C++ or Python
 +
* Difficulty: Low
 +
 
 +
=== Write the One-True-Tool to unify probe/setup/configuration cifs.ko properly ===
 +
* There are too many knobs in different places at the moment: request-keys, idmap, cifscreds, /proc stuff. This goal of this project would be to write one CLI tool that would wrap everything under a common interface. It could handle getting/setting ACL as well.
 +
* This would be a userspace project.
 +
* The implementation shouldn't too hard but the student will need to learn about the existing configuration ways and usecases which can be long.
 +
* Language: C, C++, Python
 +
* Difficulty: Medium
 +
 
 +
=== Add support for DAC (Claims Based ACLs) to the smb3 kernel module and tools ===
 +
* Similar to what was done to extend the Apache access control model to allow richer ACL semantics,
 +
* SMB3's access control model was extended (at least in Windows).  See e.g. [https://docs.microsoft.com/en-us/windows/security/identity-protection/access-control/dynamic-access-control Dynamic Access Control].
 +
* Add support to the SMB3 kernel module and user space tools (or create new ones) to allow viewing and managing claims based ACLs (DAC) from the client.
 +
* Difficulty: Medium
 +
* Language: C (kernel), C/C++/Python (user space tools)
 +
 
 +
=== Add support for ODX (T10) Copy Offload to the smb3 kernel module ===
 +
* Windows, and various NAS servers support ODX copy offload (e.g. [https://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.cdot-famg-cifs%2FGUID-1323806A-F37B-46AF-B123-E40FCD362B33.html NetApp]),
 +
* to allow much faster server side copy.
 +
* Add support to the kernel client for this and integration with existing or new copy offload tools to make it easy to use. 
 +
* For more information see [https://msdn.microsoft.com/en-us/library/cc246482.aspx MS-SMB2] and [https://www.slideshare.net/gordonross/smb3-offload-data-transfer-odx SMB3 offload data transfer] and [https://docs.microsoft.com/en-us/windows-hardware/drivers/ifs/offloaded-data-transfers Offloaded data transfers] and references at bottom of [[Server-Side_Copy]].
 +
* Language: C
 +
* Difficulty: Medium/High
 +
 
 +
===VFS change notification support in cifs.ko ===
 +
* The kernel provides a file/dir notification API (inotify, dnotify). The SMB protocol also provides a way to get notified of file changes. This project would be about implementing the inotify API for cifs.ko by making use of the SMB notification mechanism.
 +
* Difficulty: Hard
 +
* Language: C
 +
 
 +
===Failover/Continuous Availability and HA improvements (Witness protocol)===
 +
* Benefits: Improved reliability, data integrity - may also allow planned migrations (moving data from one server to another). 
 +
* Challenges: Complexity, requires additional RPC infrastructure in client.  There is a Samba user space prototype of the Witness protocol that could be reused (since we only need the client part of the RPC calls).
 +
* Language: C
 +
* Difficulty: High
 +
 
 +
<!-- Commenting out stale proposals
  
*Difficulty: Easy
+
===File Copy Offload: T10 operations, and improved tools for using CopyChunk===
*Language(s): C
+
* Benefits: Improved performance.  Copy offload is useful for quickly replicating large files, and for backup and for virtualization. Good news is that one copy offload mechanism (CopyChunk) already works.  Windows 2012 introduced a [https://msdn.microsoft.com/en-us/library/windows/desktop/hh848056(v=vs.85).aspx second mechanism] and also see pages 33 to 42 of [http://www.snia.org/sites/default/files/SNIA_SMB3_final.pdf SNIA SMB3]).  May be even more useful if TRIM/DISCARD support also added.  This is also very timely given the recent support in the linux kernel vfs being added for the copy_range API.
*Possible mentors: Andrew Bartlett
+
* Challenges: Ensuring semantics match what is being used in the new copy_range Linux kernel interface, and if not either emulate the alternate semantics, enhance copy_range or provide additional private ioctls to handle the SMB3 copy offload semantics (CopyChunk vs. ODX)
 +
* Language: C
 +
* Difficulty: Low / moderate
 +
* Possible Mentors: Steve French
  
==Linux Kernel CIFS/SMB2 client improvements==
+
===Multiadapter support===
Interested students should contact Steve French or Jeff Layton and discuss possible improvements to the Linux Kernel CIFS VFS client. Here are some ideas to get you started:
+
* Benefits: Big performance advantage for some common cases (e.g. RSS capable adapters, and also two adapter scenarios) and prepares for RDMA in the future which will help cifs.ko in even more workloads.
 +
* Challenges: Testing may require more physical hardware (two, dual adapter machines to demonstrate performance improvements).
 +
* Language: C
 +
* Difficulty: Moderate
 +
* Possible Mentors: Steve French
  
=== SMB3 protocol improvements ===
+
===Directory oplocks===
*The SMB2 protocol (followon to cifs) and now the SMB3 protocol, new to Windows 8, Windows 2012 and Samba 4, add many useful performance enhancements and new features. SMB2.1 support, and even minimal SMB3 support, is merged into the Linux kernel client since the 3.8 kernel, but there are many useful optional features still to be implemented.  A sample list of possible features to code includes:
+
* Benefits: Will reduce network load a lot in some workloads, and improve performance as well. Works with recent Windows servers (Windows 2012 and later e.g.).
* Support for using multiple network interfaces at once under the same mount (SMB3 multichannel)
+
* Challenges: Samba does not support it yet (although this might help drive changes to the Server and Linux VFS eventually, if we have client support).
* Improved directory and metadata caching ("directory oplocks")
 
* Improved failover in clustering environments ("persistent file handles" and more generally SMB3 "continuous availability" support)
 
* File copy offload (there are multiple server side copy mechanisms possible with SMB3, T10 copy offload, SMB2/SMB3 "copy chunk" as well as an older cifs mechanism "SMB Copy" that could be implemented, compared and optimized).  This is especially timely given the improvements in Samba/btrfs integration which better optimizes SMB2/SMB3 "copy chunk" handling on the server.  This could improve file copy performance by orders of magnitude.
 
* HA availability improvements on server and/or client (the SMB3 "Witness protocol")
 
*
 
 
* Language: C
 
* Language: C
* Difficulty: Varies, Medium to Hard, but most of the protocol features are at least well documented in WSPP and have implementations in current Windows clients and servers already
+
* Difficulty: Moderate
 
* Possible Mentors: Steve French
 
* Possible Mentors: Steve French
  
 
=== Support for SELinux ===
 
=== Support for SELinux ===
* Mac Security Label support is important for virtualization and useful for improved security some workloads.  Support for setting/getting these labels over the wire was investigated in the NFS version 4 workgroup.  Adding support to the CIFS Unix Extensions (Linux kernel client and Samba server) should be possible, especially if this is just a new class of extended attribute.  The goal would be to support this feature of SELinux to allow KVM and other applications to take advantage of security labels.  Some of the background requirements are loosely related to the (nfs equivalent of) what is mentioned in:  http://tools.ietf.org/html/draft-quigley-nfsv4-sec-label-01
+
* Mac Security Label support is important for virtualization and useful for improved security some workloads.  Support for setting/getting these labels over the wire was investigated in the NFS version 4 workgroup.  Adding support to the CIFS Unix Extensions (Linux kernel client and Samba server) should be possible, especially if this is just a new class of extended attribute.  The goal would be to support this feature of SELinux to allow KVM and other applications to take advantage of security labels.  Some of the background requirements are loosely related to the (nfs equivalent of) what is mentioned in:  [http://tools.ietf.org/html/draft-quigley-nfsv4-sec-label-01 NFSv4]
 
* Language: C
 
* Language: C
 
* Difficulty: Hard
 
* Difficulty: Hard
Line 113: Line 259:
  
 
===Create GUI or command-line tools for displaying /proc/fs/cifs statistics and and mount/session status===
 
===Create GUI or command-line tools for displaying /proc/fs/cifs statistics and and mount/session status===
* might also involve some cleanup of the in-kernel stats / status output
+
* Might also involve some cleanup of the in-kernel stats / status output.
 +
* A mostly complete [http://oss.sgi.com/archives/pcp/2013-08/msg00090.html cifs.ko Performance Co-Pilot (PCP) monitoring agent] was implemented in 2013.
 
* Language: some C (for kernel code), something else for GUI?
 
* Language: some C (for kernel code), something else for GUI?
 
* Difficulty: Easy
 
* Difficulty: Easy
Line 122: Line 269:
 
* add a way for the client to remap the uids returned by the server to uids which would be valid on the client (or to a default if such uid does not exist).
 
* add a way for the client to remap the uids returned by the server to uids which would be valid on the client (or to a default if such uid does not exist).
 
* This is helpful especially when the server supports the CIFS Unix Extensions and has different uids and gids mapping than the client
 
* This is helpful especially when the server supports the CIFS Unix Extensions and has different uids and gids mapping than the client
* Difficulty: Hard
 
* Possible Mentors: Steve French
 
 
===VFS change notification support===
 
* add VFS support for calling into the filesystem when setting up notifications
 
* add code to cifs/smb2 to set up and deal with notifications from the server in response to inotify/dnotify calls
 
 
* Difficulty: Hard
 
* Difficulty: Hard
 
* Possible Mentors: Steve French
 
* Possible Mentors: Steve French
Line 136: Line 277:
  
 
===cifs->Samba automated test facility===
 
===cifs->Samba automated test facility===
* Do build verification similar to what we can now do with the Samba server and tools in the Samba build farm.  Mounts from the Linux SMB2 and CIFS kernel clients could be tested with posix file i/o tests which might include modified versions of the "connectathon" and xfstest test suites and others.  The goal is to quickly identify problems with newly integrated patches.
+
* Do build verification similar to what we can now do with the Samba server and tools in the Samba build farm.  Mounts from the Linux SMB3, SMB2 and CIFS kernel clients could be tested with posix file i/o tests which might include modified versions of the "connectathon" and xfstest test suites and others.  The goal is to quickly identify problems with newly integrated patches by running automatically against a variety of cifs/smb2/smb3 mounts (and mount options) to ensure that regressions aren't introduced.
* Difficulty: Hard
+
* xfstests support for CIFS was added as part of [[SoC/2014]].
 +
* Difficulty: Medium
 
* Possible Mentors: Steve French
 
* Possible Mentors: Steve French
  
 
===Other Random Ideas===
 
===Other Random Ideas===
 
* Ideas aren't limited to these, feel free to propose something else:
 
* Ideas aren't limited to these, feel free to propose something else:
 +
** Improve integration between cifs.ko and userspace Samba tools and libraries.  Allow userspace Samba libraries to use an existing CIFS mount if it exists by passing requests (via an ioctl or other user->kernel IPC) to cifs.ko.  This could improve performance but also more naturally allow use of the same credentials for a user across file and management operations (e.g. listing shares via smbclient and mounting that share).
 
** Create a GUI for creating and managing Linux cifs mounts, and more easily configuring the many complex cifs mount options, statistics (/proc/fs/cifs)
 
** Create a GUI for creating and managing Linux cifs mounts, and more easily configuring the many complex cifs mount options, statistics (/proc/fs/cifs)
 
** Support for alternate transport protocols (other than TCP sockets).  Adding support for SCTP to cifs/smb2 kernel clients and Samba server or perhaps more interesting add support for Linux's "virtio" transport to the cifs/smb2 kernel clients and Samba server (to allow optimized mounts and zero-copy transfer of data from virtualized guests to hosts on the same box)
 
** Support for alternate transport protocols (other than TCP sockets).  Adding support for SCTP to cifs/smb2 kernel clients and Samba server or perhaps more interesting add support for Linux's "virtio" transport to the cifs/smb2 kernel clients and Samba server (to allow optimized mounts and zero-copy transfer of data from virtualized guests to hosts on the same box)
Line 147: Line 290:
 
** Add additional library support or modify Samba client libraries so they can use existing kernel cifs functions (such as sending SMBs on negotiated sessions when the kernel client already has a session to the server).  With the addition of library to access cifs's pipe (in kernel), Samba client libraries or other dce/rpc code could use cifs kernel sessions for management of and over cifs mounts.
 
** Add additional library support or modify Samba client libraries so they can use existing kernel cifs functions (such as sending SMBs on negotiated sessions when the kernel client already has a session to the server).  With the addition of library to access cifs's pipe (in kernel), Samba client libraries or other dce/rpc code could use cifs kernel sessions for management of and over cifs mounts.
 
** Add libraries and utilities to manage acls (cifs kernel client has an extended attribute for setting/getting "raw" cifs acls but userspace posix acl tools obviously can't be used to manage cifs specific acl features).
 
** Add libraries and utilities to manage acls (cifs kernel client has an extended attribute for setting/getting "raw" cifs acls but userspace posix acl tools obviously can't be used to manage cifs specific acl features).
*Difficulty: Varies
+
*Difficulty: Low
 
*Language(s): C
 
*Language(s): C
 
*Possible mentors: Steve French
 
*Possible mentors: Steve French
Line 167: Line 310:
 
*Difficulty: Easy to Medium
 
*Difficulty: Easy to Medium
 
*Language(s): HTML, CSS, Python
 
*Language(s): HTML, CSS, Python
*Possible mentors: Matthieu Patou, [[JelmerVernooij]]
+
*Possible mentors: Matthieu Patou
 
+
-->
== Samba GTK+ ==
 
 
 
[[Samba-GTK Samba-GTK]] is a set of GTK+ frontends for Samba written in Python.
 
 
 
===Extension of the GTK+ frontends=== 
 
  
There are a couple of GTK+ frontends for Samba4 (see [[SambaGtk]]). These are very limited at the moment but you could work on expanding them and further integrating them with GNOME. Language: C or Python
+
== Wireshark ==
 +
Wireshark has 2 SMB dissectors: "smb" for SMB1, "smb2" for SMB2 and above. It also has a DCE/RPC ([https://en.wikipedia.org/wiki/Microsoft_RPC MSRPC]) dissector that is generated from Samba IDL files.
  
*Difficulty: Easy
+
=== Improve smbcmp, the capture diff tool ===
*Language(s): Python, perhaps C
+
* [https://github.com/aaptel/smbcmp smbcmp] is a simple CLI tool that uses Wireshark CLI version (tshark) to dump and diff traces. It currently uses the plain text output format of tshark but tshark also has a proper XML output. The goal of this project would be to use or combine current tshark output with the XML output to do better and deeper diffs (ignoring indentation differences, adding ways to let users add ignore rules, etc).
*Possible mentors: [[JelmerVernooij]]
+
* Make smbcmp highlight diffs from the packet summary listing (currently it only diffs the "detail packet" output).
 
+
* These are just ideas, students are free to propose and work on their own ideas.
=== Port to GTK3 ===
+
* Language: Python (a rewrite in something else is OK too)
 
+
* Difficulty: Low
*Difficulty: Easy
+
* Possible Mentors: Aurélien Aptel
*Language(s): Python
 
*Possible Mentors: [[JelmerVernooij]]
 

Latest revision as of 09:10, 15 March 2019

Google Summer of Code: Suggested Project ideas

The following are the Samba project ideas for Summer of Code. Of course you are free to come up with ideas not listed here. Please discuss the your planned project by either joining us on irc://irc.freenode.net/#samba-technical or by sending email to samba-technical@lists.samba.org

Samba

Some additional possible GSoC topics can be found in Bugzilla in the form of bugs which are marked as "Feature request": here. Questions regarding complexity and requirements should be directed to the technical mailing list.

Print System Asynchronous Remote Protocol Wireshark Dissectors

The Print System Asynchronous Remote Protocol (MS-PAR) is a replacement for the synchronous Print System Remote Protocol (MS-RPRN). MS-PAR inherits many message and buffer formats from the old protocol, but allows for asynchronous submission and notification of print jobs. Further details of the protocol can be found in Günther and Andreas' SambaXP presentation.

The student should write Wireshark dissectors for MS-PAR. The student should improve existing smbtorture tests, which demonstrate how the protocol works against a Windows server.

  • Difficulty: Medium
  • Language(s): C
  • Possible Mentors: Andreas Schneider (supported by Günther Deschner)

dbwrap back-end for Ceph RADOS key-value storage

Ceph offers a highly scalable and fault-tolerant storage system. Samba is already capable of sharing data located on the Ceph Filesystem, however scale-out sharing (the same data exposed by multiple Samba nodes) currently requires the use of CTDB for consistent and coherent state across Samba cluster nodes. In such a setup CTDB provides a clustered database with persistent key-value data storage and locking. Database usage is abstracted out via a generic dbwrap interface.

Ceph's librados library provides an API for the storage and retrieval of arbitrary key-value data via the omap functions. A watch/notify protocol is also provided as a mechanism for synchronising client state (locking). Key-value data stored in the RADOS back-end inherits the same redundancy features as regular objects, making it a potentially good candidate as a replacement for CTDB in scale-out Samba clusters.

This task involves the implementation and testing of a new dbwrap back-end that uses librados for the storage, retrieval and locking of Samba key-value state. Ideally, the candidate would also allow time for benchmarking.

  • Difficulty: Medium
  • Language(s): C
  • Possible Mentors: David Disseldorp


Samba binary size reduction

Samba has grown to quite a bloated beast. This task will focus on some areas where the bloat can easily be reduced, e.g.

  • Removal of unused autogenerated librpc code
    • RPC client and server code when only serialization functions are used
    • optional struct print routines (pidl noprint?)
    • Some knowledge of perl would help here (for pidl)
  • Add new build options to compile Samba without certain functionality
    • Undesired DCE/RPC services
    • Printing support
    • Legacy SMB/CIFS support (stretch goal)


  • Difficulty: easy
  • Language(s): Some knowledge of C and python would be helpful
  • Possible Mentors: David Disseldorp (supported by Andrew Bartlett)

Fuzz Samba using American Fuzzy Lop

Try to find bugs in the server or client components of Samba using AFL. This is not the usual AFL scenario where you can fuzz a file format parser. It needs to be a bit smart about it, as the way the SMB protocol works, you need to do a series of steps before reaching arbitrary SMB commands (protocol negotiation, session setup, tree connect, ...). Similarly some checks like packet signing should be worked around to reach the best results from AFL.

Samba code will need to be modified in hackish ways to make this work. I don't expect to be able to merge it back but if some of the modification required could be cleaned up and integrated in would be a bonus.

  • Difficulty: medium
  • Language(s): Some knowledge of C
  • Possible Mentors: Aurélien Aptel

Samba AD DC as the ideal POSIX Directory

Samba is a great Active Directory Domain Controller, but it is not an ideal directory server for a large, passionate and important user base: Sites with Samba SMB servers, but also general purpose Linux servers. A smaller subset of these sites also have Linux desktops. These sites may also have Windows servers, but they like the Windows desktops, are not the focus.

These sites often used Samba + OpenLDAP, and are finding the move to Samba's AD DC a bit difficult, because schema extension is hard, some things are not done automatically (like uidNumber allocation), and in general the focus has been around matching Windows not listening to the needs of this part of our user base.

Specific research should be done into what FreeIPA does well in targeting this user segment, and what customisations advanced users of OpenLDAP apply.

This project would be to propose a number of specific improvements, and to add both tests and an implementation of these improvements to Samba.

  • Difficulty: Hard
  • Languages(s): C, Python
  • Possible Mentors: Andrew Bartlett


GitLab CI of Samba for non-linux platforms (FreeBSD in particular)

Samba uses GitLab CI to improve the quality of our patches. Efforts are currently underway to extend the docker container used from just Ubuntu 14.04 to later versions and other distributions.

However, we occasionally have issues ensuring Samba still builds and operates on FreeBSD and other non-linux platforms.

The idea would be to have a docker image and .gitlab-ci.yml code to support it that runs FreeBSD and then builds and runs Samba's testsuite inside that FreeBSD nested VM, while still outputting the results to the normal gitlab-ci.

This differs from just running GitLab CI runners on FreeBSD as we need auto-scale, destroy the host and guest at the end of the test and run on Linux docker (such as the free GitLab,com CI runners).

As a stretch goal, being able to run some tests against a specific linux kernel and a raw ext4 filesystem (rather than unionfs) via qemu rather than depending on the docker host configuraiton.

  • Difficulty: Medium
  • Language(s): Python, shell, YAML
  • Possible Mentors: Andrew Bartlett

Integrate Samba AD DC deployment and management with Cockpit

A prototype at https://github.com/abbra/cockpit-app-samba-ad shows how we can integrate Samba AD deployment with Cockpit Linux management console. A goal of this task is to move forward with this prototype to produce a fully working Samba AD management tool for common operations supported by 'samba-tool' command line utility.

  • Difficulty: Medium
  • Language(s): Python, JavaScript (React), CSS, HTML
  • Possible Mentors: Alexander Bokovoy


Linux Kernel SMB Client Improvements

The Linux Kernel has a module called cifs.ko which is independent from Samba (it doesn't share code) that allows users to mount remote shares. It supports multiple dialects of SMB (1, 2, 3). The protocol dialects are now officially documented by Microsoft (See MS-SMB, MS-SMB2) so students shouldn't have to worry about reverse engineering to understand them. The Wireshark open source network sniffer&dissector is a very good learning tool as well.


Interested students should contact Steve French or the linux-cifs mailing list to discuss possible improvements to the Linux Kernel CIFS VFS client.

Add machine-readable debug & stats /proc file

  • We currently output debug and statistic information under /proc/fs/cifs/ (DebugData, Stats, ...). We need to stop outputing free format text that breaks all parsers out there everytime we add things to it. Clean up the cifsdebug.c file (it is kind of messy). Possibly generate a hierarchy of /proc files (e.g. a dir per tcp connection, subdirs for session, files for tcons) instead of dumping everything in one file.
  • Make a nice visualizer/dashboard thing to get an overview. This could be console or GUI or...
  • Language: C for the kernel stuff, Userspace can be C, C++, Python.
  • Difficulty: Low

Add additional ftrace (trace-cmd) tracepoints and better GUI

  • Add more dynamic tracepoints to cifs.ko for commonly needed user scenarios, and add GUI (and/or CLI) tool to make it easier to enable/disable cifs.ko dynamic trace points (See /sys/kernel/debug/tracing/events/cifs/ for the pseudo-files that are currently configured manually for tracing or via trace-cmd) Make a nice native/console/web UI for it.
  • Language: C (any kernel changes) and userspace C, C++ or Python
  • Difficulty: Low

Add performance analysis cli tools

  • Add more perf tools for SMB3 client (similar to iostat or nfsstat) that leverage (and possibly extend what is captured) in /proc/fs/cifs/Stats but make it easier to analyze performance of a cifs mount
  • Language: C, C++ or Python
  • Difficulty: Low

Write the One-True-Tool to unify probe/setup/configuration cifs.ko properly

  • There are too many knobs in different places at the moment: request-keys, idmap, cifscreds, /proc stuff. This goal of this project would be to write one CLI tool that would wrap everything under a common interface. It could handle getting/setting ACL as well.
  • This would be a userspace project.
  • The implementation shouldn't too hard but the student will need to learn about the existing configuration ways and usecases which can be long.
  • Language: C, C++, Python
  • Difficulty: Medium

Add support for DAC (Claims Based ACLs) to the smb3 kernel module and tools

  • Similar to what was done to extend the Apache access control model to allow richer ACL semantics,
  • SMB3's access control model was extended (at least in Windows). See e.g. Dynamic Access Control.
  • Add support to the SMB3 kernel module and user space tools (or create new ones) to allow viewing and managing claims based ACLs (DAC) from the client.
  • Difficulty: Medium
  • Language: C (kernel), C/C++/Python (user space tools)

Add support for ODX (T10) Copy Offload to the smb3 kernel module

  • Windows, and various NAS servers support ODX copy offload (e.g. NetApp),
  • to allow much faster server side copy.
  • Add support to the kernel client for this and integration with existing or new copy offload tools to make it easy to use.
  • For more information see MS-SMB2 and SMB3 offload data transfer and Offloaded data transfers and references at bottom of Server-Side_Copy.
  • Language: C
  • Difficulty: Medium/High

VFS change notification support in cifs.ko

  • The kernel provides a file/dir notification API (inotify, dnotify). The SMB protocol also provides a way to get notified of file changes. This project would be about implementing the inotify API for cifs.ko by making use of the SMB notification mechanism.
  • Difficulty: Hard
  • Language: C

Failover/Continuous Availability and HA improvements (Witness protocol)

  • Benefits: Improved reliability, data integrity - may also allow planned migrations (moving data from one server to another).
  • Challenges: Complexity, requires additional RPC infrastructure in client. There is a Samba user space prototype of the Witness protocol that could be reused (since we only need the client part of the RPC calls).
  • Language: C
  • Difficulty: High


Wireshark

Wireshark has 2 SMB dissectors: "smb" for SMB1, "smb2" for SMB2 and above. It also has a DCE/RPC (MSRPC) dissector that is generated from Samba IDL files.

Improve smbcmp, the capture diff tool

  • smbcmp is a simple CLI tool that uses Wireshark CLI version (tshark) to dump and diff traces. It currently uses the plain text output format of tshark but tshark also has a proper XML output. The goal of this project would be to use or combine current tshark output with the XML output to do better and deeper diffs (ignoring indentation differences, adding ways to let users add ignore rules, etc).
  • Make smbcmp highlight diffs from the packet summary listing (currently it only diffs the "detail packet" output).
  • These are just ideas, students are free to propose and work on their own ideas.
  • Language: Python (a rewrite in something else is OK too)
  • Difficulty: Low
  • Possible Mentors: Aurélien Aptel