WSP

WSP (Windows Search Protocol) support in samba

support in samba

• since samba-4.20 samba ships a command line client for searching using the WSP protocol. The 'wspsearch' cli client does not work against a samba server as it currently does not implement the WSP protocol

WSP server support

• The WSP protocol is now supported in upstream however it isn't in any released branches yet (should hit samba-4.26.0)

what works with windows clients ?

searching by kind (from dropdown in search view)

searching by phrase (entered into search bar)

Matches for search phrase are also highlighted if the associated content summary contains the search phrase

folder search results

Recently support for folder results has been added. Although it has always been possible to search the both the fscrawler main and folder indexes, it wasn't until recently possible to distinguish whether a match was a folder or file result. This is now possible with later versions of fscrawler. By setting the 'elasticsearch:wsp use fscrawler folders' smb.conf parameter to true folder results are now included in any searches performed.

can I try it out ?

Yes you can, the simplest way to try it out is to use docker compose to run a test setup with practically zero configuration or interaction. The compose setup is also ideal for testing, experimentation or just getting used to the various pieces that need to be configured and setup. Please see https://gitlab.com/npower/wsp-container Note: This project is not intended for production use but for testing and experimentation only.

Alternatively if you are willing you can build from the master branch

  git clone git@gitlab.com:samba-team/devel/samba.git
  cd samba
  ./configure.developer # (and install all the dependencies)
  make
  sudo make install

WSP running and testing using elasticsearch

Note these instructions are for a developer environment, for ease of debugging and configuration security features will be disabled.

install elasticsearch

using elasticsearch-9.5.2 (latest version at time of writing)

rpm -ivh elasticsearch-9.5.2-x86_64.rpm

take note of the generated built-in superuser 'elastic' (output as part of the rpm install)

if desired change the generated superuser password (note: elasticsearch service needs to be running)

/usr/share/elasticsearch/bin/elasticsearch-reset-password -iu elastic

start it

systemctl daemon-reload
systemctl start elasticsearch.service

check if is running

systemctrl status elasticsearch.service

check communication

 curl -k -uelastic:elastic https://127.0.0.1:9200

should respond with

{
 "name" : "localhost.localdomain",
 "cluster_name" : "elasticsearch",
 "cluster_uuid" : "n-AXwOQeTOSddb_p3UXsUQ",
 "version" : {
   "number" : "8.15.2",
   "build_flavor" : "default",
   "build_type" : "rpm",
   "build_hash" : "98adf7bf6bb69b66ab95b761c9e5aadb0bb059a3",
   "build_date" : "2024-09-19T10:06:03.564235954Z",
   "build_snapshot" : false,
   "lucene_version" : "9.11.1",
   "minimum_wire_compatibility_version" : "7.17.0",
   "minimum_index_compatibility_version" : "7.0.0"
 },
 "tagline" : "You Know, for Search"
}

configure elasticsearch

Note: This is a developer setup, not suitable for production, please refer to the elasticsearch documentation for specific information about securing elasticsearch.

edit /etc/elasticsearch/elasticsearch.yaml

 xpack.security.enabled: false
 xpack.security.http.ssl:
   enabled: false
 xpack.security.transport.ssl:
   enabled: false

restart elasticsearch

systemctl restart elasticsearch.service

WSP running and testing using opensearch (version dependant, will NOT work post opensearch >= 3.0

install opensearch

using opensearch-2.19.6 (latest version at time of writing)

OPENSEARCH_INITIAL_ADMIN_PASSWORD=1234?Changeme rpm -ivh opensearch-2.15.0-linux-x64.rpm

start it

systemctl daemon-reload
systemctl start opensearch.service

check if is running

systemctl status opensearch.service

check communication

 curl -k -uadmin:1234?Changeme https://127.0.0.1:9200

should respond with

{
 "name" : "localhost.localdomain",
 "cluster_name" : "opensearch",
 "cluster_uuid" : "6fJA5WMmSiK2wc4rHdkVvw",
 "version" : {
   "number" : "7.10.2",
   "build_type" : "rpm",
   "build_hash" : "61dbcd0795c9bfe9b81e5762175414bc38bbcadf",
   "build_date" : "2024-06-20T03:27:31.591886152Z",
   "build_snapshot" : false,
   "lucene_version" : "9.10.0",
   "minimum_wire_compatibility_version" : "7.10.0",
   "minimum_index_compatibility_version" : "7.0.0"
 },
 "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

configure opensearch

Note: This is a developer setup, not suitable for production, please refer to the opensearch documentation for specific information about securing opensearch

• disable ssl, user authentication etc.

for testing it is convenient to be able to easily see the communication between samba and opensearch unencrypted, of course ssl can be re-enabled after a working setup has been established.

in /etc/opensearch/opensearch.yml

plugins.security.disabled: true

• allow fscrawler to talk to opensearch

Add following line to /etc/opensearch/opensearch.yml

compatibility.override_main_response_version: true (to allow fscrawler to communicate with opensearch)

restart opensearch

systemctl restart opensearch.service

Use fscrawler to index files for elasticsearch or opensearch

using latest fscrawler snapshot version 3.0 (at time of writing)

• identify (or create) some locations on the filesystem (which are accessible from samba shares) that have content you would like to index
• install fscrawler

unzip fscrawler-3.0.zip

• create a user to use to communicate with elasticsearch or opensearch to populate the index

Here we will use the 'elastic' user that comes already setup with elaticsearch. Note: the elasticsearch 'elastic' user is a super user. You might want to consider creating a specific elasticsearch user for fscrawler to use that has appropriate roles assigned. e.g. with 'just enough' privileges to access the index(s) you want to create/modify. Same applies to opensearch

for elasticsearch see creating users, creating roles, creating roles, creating API key similarly for opensearch see here and associated documentation.

• use fscrawler to create an index

 ./fscrawler-distribution-3.0/bin/fscrawler --setup index_name

• edit the config file ~/.fscrawler/index_name/_settings.yaml created in the last step

example of a minimal config (without ssl) For details on defaults please see the comments in the content of the _settings.yaml created in the above step

name: "index_name"
fs:
  url: "/some/path/"
  # optional: if false, the crawler will stop crawling at the first error found
  continue_on_error: true
  # you may want to store (partial) content of the file (see indexed_chars)
  index_content: true
  # include size ot file in index
  add_filesize: true
  # include user/group of file only if needed
  attributes_support: true
  raw_metadata: true
  # optional: recommended, but will create another index
  index_folders: true
elasticsearch:
  # optional: list of Elasticsearch node urls
  urls:
    - "http://127.0.0.1:9200"

• run fscrawler again

./fscrawler-distribution-3.0/bin/fscrawler index_name --loop 1

Configure WSP for samba

use the following global configuration

wsp backend = elasticsearch

use the following share configuration

wsp = true
elasticsearch:index = index_name
elasticsearch:max results = 200
elasticsearch:wsp use fscrawler folders = true

start samba

systemctrl start smb.service

wsp smb.conf configuration parameters

non WSP specific parameters

name	value	description
elasticsearch:address (S)	needle.haystack.samba.org	Specifies the name of the Elasticsearch server to us
elasticsearch:index (S)	index	name of elasticsearch index to search
elasticsearch:max results (S)	100	number of results to return
elasticsearch:port (S)	8000	TCP port of Elasticsearch server to use
elasticsearch:use tls (S)	no	A boolean value with specifies whether to use HTTPS when talking to the Elasticsearch server

WSP specific parameters

name	value	description
elasticsearch:wsp use fscrawler folders (S) [*]	yes	A boolean specifying whether to additionally use fscrawler's folder index to include folder results in searches.
elasticsearch:wsp_acl_filtering (S) [*]	false	A boolean specifying if results should be filtered based on whether the smbclient user has access to the result
elasticsearch:wsp_mappings (G) [*]	mymappings.json	A json file specifying the mapping of elasticsearch attributes to WSP properties (and associated conversions)
wsp backend (G) [*]	elasticsearch	string specifying the supported backend (elasticsearch currently the only supported backend)

[*] These parameters are not yet in the 'official' samba sources

use wspsearch cli or windows client to search for content (e.g. pictures)

wspsearch -U$user%$password //$host/$share --kind picture

Securing indexed content

It is important to understand that the samba WSP server only translates the WSP protocol messages that make up a query and/or traversal and retrieval of results into requests that elastic/opensearch server can understand. Those requests are made against elastic/opensearch using either the anonymous user or hopefully in the future also a basic user or an api key (only for elasticsearch)

samba WSP communicates with elastic/openseach using encrypted (or plain text) http connections using anonymous access. (and planned in the future basic authentication).

populating the index is a separate process, fscrawler is a tool/process for scanning files (of different types e.g. videos, photographs, documents etc.) and populating an elastic/opensearch index with the metadata that it has extracted from those files. In order to access the files fscrawler of course needs to run as a user that has permission to read those files. The fscrawler configuration can use anonymous access or use elastic/opensearch credentials defined that will be used to communicate with the elastic/opensearch instance or those credentials can be passed on the command line

so the users involved in searching and retrieving results are

• the local user on the host machine (where the files to be indexed are located) that fscrawler process runs as to read the files to produce meta data which will be used to populate the index
• the elastic/opensearch user used by fscrawler to populate the elastic/opensearch index (currently anonymous)
• the user used by the samba WSP server to query the elastic/opensearch index (currently anonymous)
• the authenticated samba user to perform the search (who may or maynot have permission to access the filenames returned by the search)

There is therefore a likely disconnect between the user used to populate the index, the user used to search that index and the authenticated samba user.

The samba WSP server by default will acl check the files in the results and filter out results that the authenticated samba user cannot read, this can be disabled with the following setting

   elasticsearch:wsp_acl_filtering [S]

acl filtering brings with it some penalties

• query results are now cached,
• you MUST specify a limit on the number of results returned (cached)

elasticsearch:max results [S] or the query will fail.

• depending on the data stored in the indexes searches could be take a much greater time than expected, for example a search might yield a large amount of results but if the results that satisfy the acl check are mostly the end of the result set then a very large amount of results (or maybe even all the results) will need to be tested before the 'max results' to be cached is reached
• because results are now cached the memory footprint will be impacted and many concurrent searches could affect available memory of the host system

If at all possible it is better to NOT enable acl filtering. That way the search results don't need to be additionally filtered however the flip side is

• filenames and paths not readable to the authenticated samba user MAY be returned as results of a search (possible information leak)
• similarly information such as keywords or content from a document result may be exposed to a user that otherwise might not have permission to access that information

(FUTURE WORK, not currently implemented) Using elastic/opensearch to limit results

The user to authenticate against elastic/opensearch could be specified by the mapping between the authenticated samba user and an elastic/opensearch basic user.

elastic/opensearch users have roles associated with them, those roles can be configured to limit or allow access to specific indexes. So, it could be possible to limit access to the information stored in an index based on the role associated with the authenticated elastic/opensearch user. The authenticated elastic/opensearch user to be used to communicate with elastic/openseach could be mapped to the authenticated samba user (see above)

Added granularity can be added by using index names that are user specific, e.g the index name configured to used by samba when sending a query to elasticsearch can use variable substitutions so the index name could for example be based on a variation of the username of the authenticated samba user (this is suitable for an index dedicated to specific share that is a personal data store for that user)

the role associated with the authentication elastic/opensearch user can also be modified to use field level security which allows sensitive fields to included or excluded from the results of a query

elasticsearch:

• see document level security
• see field level security

opensearch:

• see document level security
• see field level security

Additionally to align the users that have access to a share (and it's files) you could use a combination of 'force user', 'valid groups' or 'valid users' to limit access to shares to be searched with elastic/opensearch. This for example could be used to ensure there is a match between the files 'crawled' using a user with a certain group and the group used to access the share (and it's files).