WSP: Difference between revisions
(Created page with "=WSP running and testing using opensearch= ==install opensearch== <code> OPENSEARCH_INITIAL_ADMIN_PASSWORD=1234?Changeme rpm -ivh opensearch-2.15.0-linux-x64.rpm </code> ==start it== systemctl daemon-reload systemctl start opensearch.services ==check if is running== systemctrl status opensearch.services ==check communication== curl -k -uadmin:1234?Changeme https://127.0.0.1:9200 should respond with { "name" : "localhost.localdomain", "cluster_name" : "opensea...") |
No edit summary |
||
Line 1: | Line 1: | ||
=WSP running and testing using opensearch= |
=WSP running and testing using opensearch= |
||
==install opensearch== |
==install opensearch== |
||
using opensearch-2.15.0 (latest version at time of writing) |
|||
<code> |
<code> |
||
OPENSEARCH_INITIAL_ADMIN_PASSWORD=1234?Changeme rpm -ivh opensearch-2.15.0-linux-x64.rpm |
OPENSEARCH_INITIAL_ADMIN_PASSWORD=1234?Changeme rpm -ivh opensearch-2.15.0-linux-x64.rpm |
||
Line 31: | Line 33: | ||
'''Note:''' This is a developer setup, not suitable for production, please refer to the opensearch documentation for specific information about securing opensearch |
'''Note:''' This is a developer setup, not suitable for production, please refer to the opensearch documentation for specific information about securing opensearch |
||
* disable ssl |
* disable ssl |
||
::for testing it is convenient to be able to easily see the communication between samba and opensearch unencrypted, of course ssl can be |
::for testing it is convenient to be able to easily see the communication between samba and opensearch unencrypted, of course ssl can be re-enabled after a working setup has been established. |
||
in /etc/opensearch/opensearch.yml: |
in /etc/opensearch/opensearch.yml: |
||
-plugins.security.ssl.http.enabled: false |
-plugins.security.ssl.http.enabled: false |
||
Line 39: | Line 41: | ||
compatibility.override_main_response_version: true (to allow fscrawler to communicate with opensearch) |
compatibility.override_main_response_version: true (to allow fscrawler to communicate with opensearch) |
||
==Use fscrawler to index files== |
==Use fscrawler to index files== |
||
using latest fscrawler version 2.10 (at time of writing) |
|||
⚫ | |||
⚫ | |||
* install fscrawler |
* install fscrawler |
||
unzip fscrawler-distribution-2.10-20240702.144319-374.zip |
unzip fscrawler-distribution-2.10-20240702.144319-374.zip |
||
* create a user |
* create a user to use to communicate with opensearch to populate the index |
||
:Here we will use the 'admin' user that comes already setup with opensearch. '''Note:''' the opensearch 'admin' user is a super user. You might want to consider creating a specific opensearch user for fscrawler to use that has appropriate roles assigned. e.g. with 'just enough' privileges to access the index(s) you want to create/modify. (see [https://opensearch.org/docs/latest/security/access-control/users-roles here] and associated documentation) |
:Here we will use the 'admin' user that comes already setup with opensearch. '''Note:''' the opensearch 'admin' user is a super user. You might want to consider creating a specific opensearch user for fscrawler to use that has appropriate roles assigned. e.g. with 'just enough' privileges to access the index(s) you want to create/modify. (see [https://opensearch.org/docs/latest/security/access-control/users-roles here] and associated documentation) |
||
* use fscrawler to create an |
* use fscrawler to create an index |
||
./fscrawler-distribution-2.10-SNAPSHOT/bin/fscrawler index_name |
./fscrawler-distribution-2.10-SNAPSHOT/bin/fscrawler index_name |
||
you will be prompted (if this is the first time to run the command) |
you will be prompted (if this is the first time to run the command) |
||
Line 87: | Line 91: | ||
*:admin%1234?Changeme |
*:admin%1234?Changeme |
||
will map all previously unmapped (in credfile) users to admin |
will map all previously unmapped (in credfile) users to admin |
||
note: as |
note: as the credentials are stored in a local file (which should be root rw only) the opensearch users defined in the credfile should have the most restrictive privileges possible (and no write permissions) |
||
==start samba== |
==start samba== |
||
systemctrl start smb.service |
systemctrl start smb.service |
Revision as of 15:17, 4 October 2024
WSP running and testing using opensearch
install opensearch
using opensearch-2.15.0 (latest version at time of writing)
OPENSEARCH_INITIAL_ADMIN_PASSWORD=1234?Changeme rpm -ivh opensearch-2.15.0-linux-x64.rpm
start it
systemctl daemon-reload systemctl start opensearch.services
check if is running
systemctrl status opensearch.services
check communication
curl -k -uadmin:1234?Changeme https://127.0.0.1:9200
should respond with
{ "name" : "localhost.localdomain", "cluster_name" : "opensearch", "cluster_uuid" : "6fJA5WMmSiK2wc4rHdkVvw", "version" : { "number" : "7.10.2", "build_type" : "rpm", "build_hash" : "61dbcd0795c9bfe9b81e5762175414bc38bbcadf", "build_date" : "2024-06-20T03:27:31.591886152Z", "build_snapshot" : false, "lucene_version" : "9.10.0", "minimum_wire_compatibility_version" : "7.10.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "The OpenSearch Project: https://opensearch.org/" }
configure opensearch
Note: This is a developer setup, not suitable for production, please refer to the opensearch documentation for specific information about securing opensearch
- disable ssl
- for testing it is convenient to be able to easily see the communication between samba and opensearch unencrypted, of course ssl can be re-enabled after a working setup has been established.
in /etc/opensearch/opensearch.yml:
-plugins.security.ssl.http.enabled: false +plugins.security.ssl.http.enabled: true
- allow fscrawler to talk to opensearch
- Add following line to /etc/opensearch/opensearch.yml
compatibility.override_main_response_version: true (to allow fscrawler to communicate with opensearch)
Use fscrawler to index files
using latest fscrawler version 2.10 (at time of writing)
- identify (or create) some locations on the filesystem (which are accessible from samba shares) that have content you would like to index
- install fscrawler
unzip fscrawler-distribution-2.10-20240702.144319-374.zip
- create a user to use to communicate with opensearch to populate the index
- Here we will use the 'admin' user that comes already setup with opensearch. Note: the opensearch 'admin' user is a super user. You might want to consider creating a specific opensearch user for fscrawler to use that has appropriate roles assigned. e.g. with 'just enough' privileges to access the index(s) you want to create/modify. (see here and associated documentation)
- use fscrawler to create an index
./fscrawler-distribution-2.10-SNAPSHOT/bin/fscrawler index_name
you will be prompted (if this is the first time to run the command)
INFO [f.console] job [index_name] does not exist INFO [f.console] Do you want to create it (Y/N)? answer 'Y'
- edit the config file ~/.fscrawler/index_name/_settings.yaml created in the last step
- configure path fscrawler to index
url: "/path/to/index"
- setup fscrawler to disable ssl when communicating with opensearch
- url: "https://127.0.0.1:9200" - url: "http://127.0.0.1:9200" - ssl_verification: true + ssl_verification: false
- don't stop on error (otherwise any problem indexing a specific file will stop the indexing process)
- continue_on_error: false + continue_on_error: true
- setup optional stuff
- attributes_support: false - raw_metadata: false + attributes_support: true + raw_metadata: true
- run fscrawler again
./fscrawler-distribution-2.10-SNAPSHOT/bin/fscrawler index_name --username admin --loop 1
configure WSP for samba
use the following global configuration
wsp backend = elasticsearch elasticsearch:auth=credfile elasticsearch:auth_file=/etc/samba/usercreds.txt elasticsearch:wsp_acl_filtering=true
use the following share configuration
wsp = true elasticsearch:index = index_name elasticsearch:max results = 200
credfile format
samba_user:opensearch_user%password
'*' can be used in place of a 'samba user' to match all currently unmatched samba users
example
*:admin%1234?Changeme
will map all previously unmapped (in credfile) users to admin note: as the credentials are stored in a local file (which should be root rw only) the opensearch users defined in the credfile should have the most restrictive privileges possible (and no write permissions)
start samba
systemctrl start smb.service
use wspsearch cli or windows client to search for content (e.g. pictures)
wspsearch -U$user%$password //$host/$share --kind picture