How to Set Up and Configure the SharePoint Online Connector

Install the Connector

Before extracting the MSI from the downloaded .zip file, select the downloaded .zip file, and check the "Unblock" box in file properties dialogue box.

1. Follow the instructions in the opening screens of the wizard.
   
2. Note where on the local machine the application is installed. Note the path.
   
3. Click Next to proceed, until you reach the "Web Service Information" screen, shown below.
   
4. DO NOT use the provided, default Domain Service Account "NetworkService Local account used by the service control manager. Not recognized by the security subsystem, so you cannot specify its name in a call to the LookupAccountName function. Has minimum privileges on the local computer and acts as the computer on the network.."
   
   1. Enter a valid local user account with admin rights on the local machine in the format domain\username
   2. Enter the account Password.
      
5. Click Next.
   
6. When finished, the wizard closes and the connector is successfully installed.
   
7. Configure the connector by adding it to your connector application: Connectivity Hub or Connector Framework, using the instructions provided in those help systems.
   
8. Specific configuration information is provided in the following topics.

Connection Configuration Settings

Set the Connection

1. After setting your generic information, click the Connect or Refresh Web Service link.

2. Select the appropriate authentication mechanism. The page refreshes and displays additional settings.

Username Authentication Settings

• User name:
  • Enter the user with the appropriate permissions for crawling the configured site collection
• Password and Application (Client) Id:
  • Enter the user password first followed by the Application ID, separated by a semi-colon (;) character. 

Certificate Authentication Settings

1. In the Certificate field, enter your Certificate common name. 

   • If you copy your certificate details, as shown in the Certificate console graphic below, your syntax will contain unnecessary spaces.
   • You must ensure you enter your certificate information in syntax without spaces or your certificate will not work.
   • CORRECT SYNTAX example: "CN=Jeff Smith"
     
   • INCORRECT SYNTAX example: "CN = Jeff Smith"
     
     
2. In the Application (Client) Id field, enter the Application ID.
   • This is a GUID such as "c87a7836-a1da-455c-9171-836bf793c195"

Common Configuration Settings

1. Disabled Security:
   1. Check to disable security.
   2. This results in no content be security-trimmed. 
2. Crawl Published Only:

   1. Check to override SharePoint settings and crawl only published documents.

   2. Any draft versions are ignored
      

3. Azure Portal Configuration Settings. Refer to the graphic below for sample values.
   1. Enter the following values in the following order, as shown in the graphic below, one per line:
      1. Tenant name
      2. Audience URI
      3. Admin URL
         

4. Subscription Id:

   1. If you are using App with Login Credentials it is possible to run the connector without the permission Sites.FullControl.All.

   2. Without this permission the connector can not perform any task which require tenant rights.

   3. The only property required to properly secure the connector is the tenant subscription ID.

      1. For the connector to secure data properly the tenant subscription ID is required.

   4. If running the connector without the permission Site.FullControl.All, the subscription ID must be provided.
      

5. Complete all remaining fields. Carefully read the instructions under each field.
   

Configure the Mailbox/Datastore Settings

Configure your mailboxes, or datastores, for the SharePoint Online v2.0 Connector, using the following steps.

• Use the Datastore Sync Settings tab to filter the datastores (site collections) that are returned after Datastore Load job Collects the logical division of your data (data stores) from the source system. Runs on a schedule Logs information according to log level set completes.

• Site Collection Title Filter:
  • Use this field to filter site collections by title.
  • You can specify both inclusion and exclusion rules.
  • One set per line.
  • Use *at beginning or end for wildcards.
  • Use the pipe character to separate multiple entries
• Site Collection Url Filter:
  • Use this field to filter site collections by URL.
  • You can specify both inclusion and exclusion rules; one set per line.
  • Use * at end for wildcards.
  • Use the pipe character to separate multiple entries.
  • Filter is case insensitive

  • The URL exclude parameter is ignored if the wildcard symbol (*) is NOT used for the include parameter
    

  • If you don’t specify the wildcard operator (*) in the include line, then the exclude option is ignored because this is a misconfiguration.

• Web Template Id Filter:
  • Use this field to filter site collections:
    • Template IDs must not include the number
      • Example:
        • SPS is accepted
        • SPS#1 is not accepted
    • You can specify both inclusion and exclusion rules; one set per line.
    • Use *at end for wildcards.
    • Use the pipe character to separate multiple entries.
    • Filter is case insensitive

Content Configuration Settings

For a given content source Content Sources do the following: Receive data from the Source System via the Connection, Filter the data it receives, Provide the results to the Target, Define the specific search index that contains the content you wish to index (and later search)., you can apply custom filters to decide which list or item to crawl. Use the following syntax for the filters:

• LISTURL =url1,url2,...
• LISTTITLE=Title1,Title2,...
• FOLDERURL=url1,url2,...   
• SITEURL=url1,url2,…

1. In the Custom Filters field, enter your filters and filter items. Separate the filters with a line break and separate the filter items with a comma (,).
2. You can exclude a filter with a hyphen (-).
   
3. Custom filters support trailing wildcards. For example, expression*.
   
4. To use a literal comma (,), precede the character with a backslash (\,).
   
5. LISTURL, LISTTITLE, FOLDERURL, SITEURL are regex compatible.
   
   • To use regex, apply the regex: expression.
     
   • For Example, LISTURL=regex:(\/Released)
     

6. If any changes are made to the Custom Filters field, a full crawl is required to properly refresh the index.

7. By default, connector returns SharePoint sites as search result items.
   
8. To exclude these entries from search results, disable Site content type using the content types filter.
   

9. To property map well known groups from SharePoint (like Everyone except external users, All Authenticated Users, and Authenticated Users), use the following Group load script:

   Group Load Script
   Copy

   dim sysId as string = HOST.GetSystemId()
   
   If sysId = "true"HOST.SetAdId("NT AUTHORITY\Authenticated Users")
   End If
   
   If sysId = "windows"HOST.SetAdId("NT AUTHORITY\Authenticated Users")
   End If
   
   If sysId.startswith("spo-grid-all-users")
   HOST.SetAdId("NT AUTHORITY\Everyone")
   End If

Complete the information in the remaining tabs using the generic instructions here:

• Connectivity Hub
  • AD Settings (Security Directory)
  • Users/Group Sync (Security: Scripts)

Extracting HTML Text and Metadata from Documents

You can extract HTML text and metadata Provides context with details such as the source, type, owner, and relationships to other data sets. Metadata provides details around the item being crawled by Connectivity Hub. from your source system Your Source System is the repository where your data is stored (data to be indexed). This repository is managed by applications such as: - SharePoint O365 - SharePoint 2013/16/19 - Documentum - File Share - OpenText - Lotus Notes - etc. Your Source System repository can also be a database such as SQL or Oracle. (optional).

• See How to Extract HTML Text and Metadata from Documents.

Extracted information includes:

• Available text and metadata which can be mapped to metadata properties. 
• HTML metadata tags from HTML pages.
• Available HTML tags and links for use as metadata from HTML documents

Update the SharePoint Online Connection with the Certificate and Azure App Information

1. Open your Connector Framework.
   
2. Click the Connections link from the top navigation.
   
3. Select your SharePoint Online v2.0 connection Connection defines the how Connectivity Hub connects to your Source System (which contains your documents, graphics, etc.,). Your Connection includes identifying elements such as: URL of the BA Insight web service connector you are using, (File Share connector, SharePoint Online connector, etc.), Authentication mode, User Accounts and Credentials, Database information (for database connectors) from the drop-down menu.
   
4. Navigate to the Certificate portion of the page.
   
5. Enter the credentials (Login and Password) of your SharePoint Online connection:
   
   1. Login
      1. Required
      2. Enter the Certificate Distinguished Name.
         
   2. Password
      1. Required
      2. Enter the Azure Application ID.

6. Click Save.