Connector Configuration
Configuration for Microsoft SharePoint Online
Please note that certain settings affect the performance of content or principal traversals, see Performance section of the FAQ for details.
Mandatory Settings
In order for the connector to connect to SharePoint Online, some actions are required on the source system as well as on the connector site. The following sections assume that the SharePoint Online related preparations described in SharePoint Online Setup have been done.
Authentication Settings
To establish a successful connection to Microsoft SharePoint Online an Azure application with certificate authentication has to be configured.
| Name | Key | Description |
|---|---|---|
Directory (Tenant) ID |
|
The ID of the Microsoft tenant. It can be found in the 'Overview' tab of the corresponding Azure application registration in the Azure Portal. |
Application (Client) ID |
|
The ID of the application registered in the Azure Portal. It can be found in the 'Overview' tab of the corresponding Azure application registration in the Azure Portal. |
Certificate File |
|
Upload for a certificate file (extension: pfx). See Certificate File Generation for a guide on how to generate a certificate. |
Certificate Password |
|
The password of the certificate file. See Certificate File Generation for a guide on how to generate a certificate. |
Optional Settings
The following sections describe which optional configuration adjustments can be done.
Principal Synchronization Settings
In various scenarios, varying strategies for principal resolution may be of interest. The connector offers support for three distinct modes:
-
Full SharePoint Online and all Azure AD groups and users:-
Fetches and resolves all SharePoint Online site collection specific groups.
-
Fetches and resolves all Entra ID users and groups.
-
-
Full SharePoint Online and referenced Azure AD groups and users-
Fetches and resolves all SharePoint Online site collection specific groups.
-
Resolves only Entra ID groups which were found as members of site collection groups.
-
-
Only SharePoint Online groups-
Fetches and resolves all SharePoint Online site collection specific groups.
-
Entra ID users and groups will not be fetched nor resolved.
-
| Resolution of SharePoint Online as well as Entra ID principals is required for secure search. This is the case, because Microsoft added Entra ID support on top of an already existing permission structure in SharePoint Online. |
| Name | Key | Description |
|---|---|---|
Principal Crawl Algorithm |
|
Defines how and in which scope the principal synchronization should be done.
Possible values are:
We recommend |
Site Discovery Settings
SharePoint Online’s top level content structure are site collections. Each site collection contains sites and lists. In order to define which site collection to traverse, the connector provides two strategies. Either a list of site collections is provided or a request against the Microsoft API will be done, fetching all site collections present in the given tenant. The result of the automatic site discovery can be narrowed down via filtering.
| Name | Key | Description |
|---|---|---|
Auto Site Discovery Enabled |
|
Defines if auto site discovery will be used. To manually state the site
collections to be traversed, unset this flag. Note: In order to use the
automatic site discovery, the Azure application requires the Microsoft Graph
permission |
Site Collections |
|
Upload for a file which contains site collections delimited by line. Only the site collections found in this file will be considered in the traversals. Needs to be configured if auto site discovery is disabled. |
General Filter Settings
The following general filter settings can be configured to restrict the traversals.
| Name | Key | Description |
|---|---|---|
Max File Size |
|
Maximal allowed size for content of documents. If the size is exceeding this value, the content of the document is discarded and ignored. |
Ignore ACL |
|
If this flag is set, the documents will be public. For public search scenarios this makes sense, because fewer requests will be executed. Furthermore, fewer permissions are required for the content traversal. |
Site Collection Filter Settings
Configuration options to restrict the traversal of site collections.
| Name | Key | Description |
|---|---|---|
Enable Site Collection URL Filtering |
|
Enables site collection filtering based on URL. |
Site Collection URL Matching Method |
|
Defines how the include / exclude entries should be matched against the site collection URL. Current supported methods are: EXACT: The site collection is filtered if its URL matches exactly with the entry. PREFIX: The site collection is filtered if its URL starts with the entry. SUFFIX: The site collection is filtered if its URL ends with the entry. SUBSTRING: The site collection is filtered if its URL contains the entry. REGEX: The site collection is filtered if its URL matches the RegEx pattern defined in the entry. |
Site Collection URL Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Site Collection URL Include List |
|
A list of URL-encoded site collection URLs. The include list entries are matched against site collection URLs. On match, the corresponding site will be traversed. |
Site Collection URL Exclude List |
|
A list of URL-encoded site collection URLs. The exclude list entries are matched against site collection URLs. On match, the corresponding site will not be traversed. |
Enable Geolocation Code Filtering |
|
Enables site collection filtering based on the geolocation code. |
Geolocation Code Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Geolocation Code Include List |
|
An include list of geolocation codes, against which all discovered site collections are evaluated via exact matching. If no include geolocation codes are configured, all site collections found will be traversed. For a list of available geolocation codes see the Official Microsoft Documentation. This list can not be configured simultaneously with the exclude list. |
Geolocation Code Exclude List |
|
An exclude list of geolocation codes, against which all discovered site collections are evaluated via exact matching. If no exclude geolocation codes are configured, all site collections found will be traversed. For a list of available geolocation codes see the Official Microsoft Documentation. This list can not be configured simultaneously with the include list. |
Site Filter Settings
Configuration options to restrict the traversed content based on site templates.
| Name | Key | Description |
|---|---|---|
Enable Site Template Filtering |
|
Enables site filtering based on templates. |
Site Template Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Included Site Templates |
|
An include list of site templates, against which all discovered site collections are matched via prefix matching.
If no include site templates are configured, all site collections found will be traversed.
The following site templates may be of interest: |
Excluded Site Templates |
|
An exclude list of site templates, against which all discovered site collections are matched via prefix matching. If no exclude site templates are configured, all site collections found will be traversed. Can not be configured simultaneously with include site templates. See Site Templates for a short guide on how to list all site templates present in your tenant. |
Enable Site URL Filtering |
|
Enables site collection filtering based on URL. |
Site URL Matching Method |
|
Defines how the include / exclude entries should be matched against the site URL. Current supported methods are: EXACT: The site is filtered if its URL matches exactly with the entry. PREFIX: The site is filtered if its URL starts with the entry. SUFFIX: The site is filtered if its URL ends with the entry. SUBSTRING: The site filtered if its URL contains the entry. REGEX: The site is filtered if its URL matches the RegEx pattern defined in the entry. |
Site URL Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
Site URL Include List |
|
A list of URL-encoded site URLs. The include list entries are matched against site URLs. On match, the corresponding site will be traversed. |
Site URL Exclude List |
|
A list of URL-encoded site URLs. The exclude list entries are matched against site URLs. On match, the corresponding site will not be traversed. |
List Filter Settings
Configuration options to restrict the traversed lists.
| Name | Key | Description |
|---|---|---|
Traverse Generic Lists |
|
When disabled, generic lists will not be traversed by the connector. |
Traverse Document Libraries |
|
When disabled, document libraries will not be traversed by the connector. |
Enable List Template ID Filtering |
|
Enables list filtering based on template IDs. For a list of standard list templates see the Microsoft Documentation. |
List Template ID Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
List Template ID Include List |
|
Defines list templates which should be traversed. This list can not be used together with the template ID exclude list. For a list of standard list templates see the Microsoft Documentation. |
List Template ID Exclude List |
|
Defines list templates which should not be traversed. This list can not be used together with the template ID include list. For a list of standard list templates see the Microsoft Documentation. |
Enable List Title Filtering |
|
Enables list filtering based on titles. |
List Title Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
List Title Matching Method |
|
Defines how the include / exclude entries should be matched against the list title. Current supported methods are: EXACT: The list is filtered if its title matches exactly with the entry. PREFIX: The list is filtered if its title starts with the entry. SUFFIX: The list is filtered if its title ends with the entry. SUBSTRING: The list is filtered if its title contains the entry. REGEX: The list is filtered if its title matches the RegEx pattern defined in the entry. |
List Title Include List |
|
Its entries are matched against list titles. On match, the corresponding list will be traversed. |
List Title Exclude List |
|
Its entries are matched against list titles. On match, the corresponding list will not be traversed. |
File Extension Filter Settings
Configuration options to restrict the traversed files based on their extension. Files which are effected by this filter are document library items and generic list item attachments.
| Name | Key | Description |
|---|---|---|
Enable File Extension Filtering |
|
Enables filtering based on file extensions. |
File Extension Filter Mode |
|
The type of filtering which should be applied. Either "INCLUDE" for including or "EXCLUDE" for excluding matching values. |
File Extension Include List |
|
The include list entries are matched against file names. Only if the file name ends with an extension mentioned in this list, the corresponding file will be traversed. |
File Extension Exclude List |
|
The exclude list entries are matched against file names. If the file name ends with an extension mentioned in this list, the corresponding file will not be traversed. |
Transcription Settings
Microsoft SharePoint Online supports transcriptions for audio and video files. By enabling this option the connector will fetch the default transcription for such files and add its metadata and content to the document library item.
(File extensions currently supported by the connector: mp4, avi, mp4v, wmv, mov)
| Name | Key | Description |
|---|---|---|
Enable Transcription Fetching |
|
If enabled the connector will embed the default transcription into document library items. |
Transcription Placement |
|
The transcription content is either placed into the item’s metadata or it will replace the binary content of the item. (Transcription metadata is always added to the library item’s metadata.) |
Metadata Settings
Configuration options to customize the metadata provided by the connector.
| Name | Key | Description |
|---|---|---|
Enable Manual Item Type Definition |
|
Enables manual definition of item types. |
Override Site Item Type |
|
Override the item type for site objects. Default: |
Override List Item Type |
|
Override the item type for list objects. Default: |
Override Document Library File Item Type |
|
Override the item type for document library files. Default: |
Override Document Library Folder Item Type |
|
Override the item type for document library folders. Default: |
Override Generic List Item Item Type |
|
Override the item type for generic list items. Default: |
Override Generic List Folder Item Type |
|
Override the item type for generic list folders. Default: |
Override Attachment Item Type |
|
Override the item type for attachments. Default: |
Connection Settings
In order to adjust the connector on HTTP client level, the following parameters can be adjusted:
| Name | Key | Description |
|---|---|---|
Socket Timeout |
|
Duration after which inactive connections will be terminated. |
Connection Timeout |
|
Maximal duration a client should wait for an initial response from a server. |
Retry Count |
|
Maximal number of times a request should be retried on failure. |
Retry Delay |
|
Fix delay between request retries. |
Max Total Connections |
|
Maximum number of allowed connections overall. |
Max Connections Per Route |
|
Number of concurrent connections per route. |
Connection Time To Live (TTL) |
|
Duration after which unused connections will be discarded. |
Requests Per Second |
|
Maximum number of requests which are active in parallel. |
Rate Limit Requests Per Second |
|
Maximum number of requests which are active in parallel while the API rate is
near its limit. |
User Agent HTTP Header |
|
HTTP header which is sent with every request. It identifies the agent responsible for the request. |
Cache Settings
Configuration options determining values for the connector’s caches.
| Name | Key | Description |
|---|---|---|
Enable Cache Stat Logging |
|
When enabled, internal cache stats (cache hit / miss rate) are logged in a given interval. |
Stat Logging Interval |
|
The interval in which the cache stats should be logged. |
Site Templates
To restrict the number of site collections fetched by auto site discovery,
the connector provides the functionality to define a set of site templates
that should be considered or not. To obtain a list of site templates used in your
SharePoint Online tenant, you can run a PowerShell script provided by Raytion.
You can find it in the scripts folder of the connector.
-
Install the SharePoint Online PowerShell module. See the official documentation from Microsoft.
-
Open a PowerShell console as an administrator and navigate to the script’s folder location.
-
Run the PowerShell script:
.\SiteCollectionsToFile.ps1
-
If the Execution Policy denies execution, it can be bypassed by using the -ExecutionPolicy parameter with the Bypass option:
powershell -ExecutionPolicy Bypass -File .\SiteCollectionsToFile.ps1
-
Enter the URL to the Microsoft SharePoint Online admin center. Format:
https://<domain>-admin.sharepoint.com -
In the appearing prompt login with a tenant administrator account.
-
Two files will be generated: one containing all the site collection URLs, and another listing all the site templates found in the tenant.
CSM Configuration
Required Configuration Properties
Optional Configuration Properties
CSM Connection Settings
Configuration options for fine-tuning the Http connection parameters.
| Name | Description |
|---|---|
Concurrent Connections |
Maximum number of concurrent open connections. |
Requests Rate |
Maximum number of requests per second. |
Connect Timeout in Milliseconds |
Timeout of the connect request. |
Socket Timeout in Milliseconds |
Timeout of the socket connected to CSM. |
Request Timeout in Milliseconds |
Timeout of a request to CSM. |
Microsoft Sharepoint Configuration
CrawlTrigger Service Settings
CrawlTrigger Service settings for fetching Submissions to SharePoint.
| Name | Property Key | Description |
|---|---|---|
Endpoint |
|
Endpoint of the CrawlTrigger Service. |
Content Source Name |
|
Name of the content source configured in SharePoint. |
Domain |
|
User Domain for the Authentication process. |
Authentication Scheme |
|
HTTP Authentication Scheme used to authenticate with the SharePoint CrawlTrigger service. Available option: NEGOTIATE. |
Username |
|
Username to authenticate to CrawlTrigger Service. |
Password |
|
Password to authenticate to CrawlTrigger Service. |
Connection Timeout |
|
Specifies the amount of time, in milliseconds, that the consumer will attempt to establish a connection before it times out. 0 is infinite. |
Receive Timeout |
|
Specifies the amount of time, in milliseconds, that the consumer will wait for a response before it times out. 0 is infinite. |
Crawl Trigger Size |
|
The size of cached Submissions to trigger a BCS Crawl. |
Crawl Interval |
|
Duration interval for checking if a crawl has to be triggered. |
Inserting Idle Duration |
|
Time duration that no other Submission is inserted to cache. After this time, a Crawl will be triggered even if the Trigger Size was not exceeded. |
Submission Repository Limit Settings (Optional)
Settings for limiting the Submission number the Repository can have, so that the used disc space is limited.
| Name | Property Key | Description |
|---|---|---|
Max. Unprocessed Submissions |
|
Maximum unprocessed Submissions inside the repository. If Submissions exceed this number then insertions are blocked until other Submissions are removed from repository. |
Max. Insertion Waiting Time |
|
Maximum insertion waiting time in case the Repository reaches the maximum size limit. After this duration the Submission will be rejected and marked as failed from the Connector. |
Repository Size Retrieval Interval |
|
Duration interval in which the connector retrieves the current number of unprocessed Submissions in the Repository. |
Submission Repository Sanitize Settings (Optional)
Settings for sanitize service of the Submission repository.
| Name | Property Key | Description |
|---|---|---|
Submission Expiration Time |
|
Max. Duration that submissions will remain in the repository in case SharePoint does not request them. |
Sanitize Interval |
|
Duration interval for cleaning up the Submission repository from expired Submissions. |
General Configuration
Database Configuration
| Name | Property Key | Description |
|---|---|---|
Configuration Type |
|
Supported are PostgreSQL, MS SQL Server, H2 and JDBC URL configuration. |
JDBC URL
| Name | Property Key | Description |
|---|---|---|
URL |
|
JDBC URL for the target database. Out of the box, the connector will use H2 file database. For productive usage, use PostgreSQL specifying the URL in format: ==== PostgreSQL Configuration |
|Setting|Description |Host |Domain name or IP address of the database server. |Port |Specifies the port number PostgreSQL is listening on, default is 5432. |Database Name |Name of the database. |Username |Username to authenticate with. The regarding user has to have read and write permissions to the database. |Password |Database Password of the configured database user.
==== Microsoft SQL Server Configuration |
|Setting|Description |Host |Domain name or IP address of the database server. Instance to connect to on server can be specified by '<server_name>\\<instance_name>' |Port |Specifies the port number MS SQL Server is listening on, default is 1433. |Database Name |Name of the database. |Add Custom Parameters |Enables the configuration of additional parameters. |Username |Username to authenticate with. The regarding user has to have read and write permissions to the database. |Password |Database Password of the configured database user.
==== H2 Configuration |
|Setting|Description
|Filename
|Name for file-based H2 database. Name must be at least three characters long and must not contain a semicolon. May include absolute or relative path to the database file.
|Add Custom Parameters
|Enables the configuration of additional parameters.
|Username
|raytion.connector.db.config.jdbc.username
| Database Username to read and write to database.
|Password
|raytion.connector.db.config.jdbc.password
|Database Password for the specified user
JDBC Url Configuration
Setting |
Description |
JDBC URL |
The database URL has to be in a valid JDBC URL format e.g. 'jdbc:postgresql//<host>:<port>/<database_name>' or 'jdbc:h2:file:<your_db_file>'. |
Username |
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
Database Password of the configured database user. |
PostgreSQL
| Name | Property Key | Description |
|---|---|---|
Host |
|
Domain name or IP address of the database server. |
Port |
|
Specifies the port number PostgreSQL is listening on, default is 5432. |
Database Name |
|
Name of the database. |
Username |
|
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
|
Password of the configured database user. |
Add Custom Parameter |
|
Enables the configuration of additional parameters. |
H2
| Name | Property Key | Description |
|---|---|---|
Filename |
|
Name for file-based H2 database. Name must be at least three characters long and must not contain a semicolon. May include absolute or relative path to the database file. |
Username |
|
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
|
Password of the configured database user. |
Add Custom Parameter |
|
Enables the configuration of additional parameters. |
MS SQL Server
| Name | Property Key | Description |
|---|---|---|
Host |
|
Domain name or IP address of the database server. Instance to connect to on server can be specified by '‹server_name>|<instance_name>'. |
Port |
|
Specifies the port number MS SQL Server is listening on, default is 1433. |
Database Name |
|
Name of the database. |
Username |
|
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
|
Password of the configured database user. |
Add Custom Parameter |
|
Enables the configuration of additional parameters. |
Oracle
| Name | Property Key | Description |
|---|---|---|
Host |
|
Domain name or IP address of the database server. Instance to connect to on server can be specified by '‹server_name>|<instance_name>'. |
Port |
|
Specifies the port number Oracle is listening on, default is 1521. |
Username |
|
Username to authenticate with. The regarding user has to have read and write permissions to the database. |
Password |
|
Password of the configured database user. |
Connection Type |
|
Specifies whether to connect to the database via the instance’s unique SID or a service name. |
Service |
|
A name of a given service / a TNS alias of the Oracle database instance. |
Traversal Configuration
| Name | Property Key | Description |
|---|---|---|
Traversal History Length |
|
Max. number of traversals to store in the history. Once the limit is exceeded, the connector will automatically remove oldest entries in the history. (default: 100) |
Include Checksum |
|
If enabled, any changes made to the pipeline e.g. configuration, the subsequent incremental run triggers a refeed of all items. |
Change Processing Interval |
|
Interval between change processing traversals. |
Resume on Start |
|
If enabled, any traversals in paused state are automatically resumed after the connector restart. Otherwise, the traversal remains in paused state. |
Number of Traversal Workers |
|
Number of workers to execute the traversal in parallel. Increasing this value might improve the performance, but will footprint higher memory consumption. It is recommended to keep the default value. (default: 10) |
Traversal Job Poll Interval |
|
Interval between the workers to be triggered to fetch and process the next tasks. (default: 10ms) |
Completion Timeout |
|
If the search engine indexes the items asynchronously, there might be some processing still in-flight during the completion process of a traversal. This value specifies the timeout value until all asynchronous callbacks are expected to return before completing the traversal. (default: 10m) |
Executor Size |
|
The executor size restricts the max. number of concurrent running traversals. |
Queue Size |
|
The queue size restricts the max. number of queued traversals. If the value is exceeded, the connector rejects further traversal requests until the queue size is below the configured size. |
Traversal Jobs
| Name | Property Key | Description |
|---|---|---|
Job Timeout Check Frequency |
|
Configures how often the connector checks for timed out jobs. |
Job Timeout |
|
The duration for which a job can stay idle before it is timed out. |
Job Cache Size |
|
Max. cache size of Jobs waiting for processing in memory. When cache is empty, next batch is fetched. |
Principal Aliaser Configuration
Principal Aliasing is applied on user information as part of Content ACL processing during Content Synchronization and Principal processing during Principal Synchronization. It’s purpose is to map external source system user to the corresponding user in search engines domain. You can configure a list of aliasers in the connector which will be applied in sequence and in order on user ACEs and user principals. The Connector supports following custom aliasing mechanism.
Custom Aliaser Disabled
If the Custom Aliaser checkbox is not selected, the connector will process user information on ACE and user principals unchanged to Search Engine. If all relevant users in the source system can be found with the same identifier in the search engine, this setup is sufficient to reflect the same secure search experience in the search engine as defined by the policy in the source system. The connector uses this option as default to process user information.
Custom Aliaser Enabled
If custom aliasing is enable then there are four types of aliaser avaialble:
Simple XML Table Aliaser
Static mapping table which can be uploaded as XML file. The connector uses the uploaded file as lookup table to map a user in the source system to a user in the search engine. Users missing a record in the file will be dropped from the ACE and during Principal Synchronization. This option is only recommended for environment with a manageable amount of users as for each user the corresponding mapping entry needs to be specified in the file.
| Name | Description |
|---|---|
XML Mapping File |
Browse and upload or drag and drop. |
Sample XML mapping file:
<?xml version="1.0" encoding="UTF-8"?>
<storeddata>
<entry keyValue="user1">user1@raytion.com</entry>
<entry keyValue="user2">user2@raytion.com</entry>
<entry keyValue="user3">user3@raytion.com</entry>
</storeddata>
Regex Replacer Aliaser
Regex Replacer Aliaser computes aliases based on a regular expression. Principals that match the regular expression are replaced by the Substitution String.
| Name | Property Key | Description |
|---|---|---|
Pattern |
|
The regular expression to match, this is the part that will be replaced. If braces (…) are used in the pattern then the matched value can be retrieved using $1 |
Substitute String |
|
String to replace the matching part of the find string. Matched value is accessed by employing $1 |
Regex Extractor Aliaser
Regex Extractor Aliaser computes aliases based on a regular expression. Principals that match the regular expression are inserted into the Insert-Into String.
| Name | PropertyKey | Description |
|---|---|---|
Pattern |
|
The regular expression to match, this is the part that will be inserted into the new value. If braces (…) are used in the pattern then the matched value can be retrieved using $$ |
Insert-Into String |
|
String to replace the matching part of the pattern. Matched value is accessed by employing $$ |
LDAP Aliaser
Ldap Aliaser searches for an LDAP entry with the requested name in the input value and returns the specified output attribute.
| Name | Property Key | Description |
|---|---|---|
Host |
|
Fully Qualified Domain Name of an LDAP server |
Port |
|
Port to use for LDAP connection, defaults are 389/636 or (recommended) 3268/3269 for simple/SSL |
AccountDN |
|
AccountDN for bind to LDAP |
Password |
|
Password part of credentials |
Input Field |
|
The Active Directory attribute name for this equality filter |
Search Root DN |
|
Distinguished Name of the subtree which is searched. The smaller the subtree the better the performance but the higher the chance of encountering principals which are not part of this subtree |
Output Field |
|
Attribute that should be returned in result entries |