How to Set Up and Configure the SharePoint 2013/16/19 Connector

All BA Insight connectors can be downloaded from Right Answers under Knowledge > BA Insight > Product Downloads > Connectors.

This connector is installed with the same generic steps as any BA Insight connector.

Satisfy the Prerequisites for your connector before installing.

Configuration specifics are detailed below.

After installing the connector, install the Site Collection Enumerator if you would like the connector to automatically enumerate the site collections rather than entering them manually.

Install and Configure the Site Collection Enumerator

  • The Site Collection Enumerator is a SharePoint solution which can be found in the folder where the SharePoint connector was installed.
  • The SharePoint solution deploys the SitesEnumerator.svc service inside the SharePoint /_vti_bin folder in the farm you wish to index.

To add the package to the SharePoint farm to index, follow these steps:

  1. Copy the SharePointSiteCollectionEnumerator.wsp package on a server of the SharePoint farm to index.
  2. Open the SharePoint Management PowerShell with admin rights and run the following command in order to add the solution to the target farm:
       Add-SPSolution "fullPathToTheWSP\SharePointSiteCollectionEnumerator.wsp"

  3. Install and deploy the Site Collection Enumerator solution:
       Install-SPSolution -Identity SharePointSiteCollectionEnumerator.wsp -GACDeployment
  4. No additional configuration is required.

Connection Configuration Specifics

Full READ permission can be given for an entire web application by way of the SharePoint Central Administration site.

You must add the user at the web app level to get full READ permissions.

  1. Login
    1. Required
    2. Enter the credentials of the account with full READ permissions.
    3. If the remote SharePoint farm to be indexed resides in an Active Directory domain not trusted by the domain with Connector Framework installed, you must also specify account credentials (in format Domain\User) to access the remote Active Directory domain.
    4. The SharePoint credentials must be added first, followed by a semi-colon and then by the remote AD domain credentials.
      For example:
        RemoteSPDomain\ReadOnlyUserName;RemoteSPDomain\ReadOnlyUserName
        ReadOnlyUserPassword;ReadOnlyUserPassword
    5. If there are multiple remote AD domains involved, add multiple credentials, each separated by a semi-colon.
  2. Site collections/Site Collections Enumerator Filter: This entry depends on the Automatically - Fetch Site Collection setting that you specify in step 11.
    If this entry is set to:
    • OFF
      • Default
      • Enter each site root URL on a new line.
      • DO NOT enter sub-site URLs.
      • For example, enter:
         https://sitecollection.sharepoint.com 
    • ON:
      •  This filter is used by the SharePoint Site Collections Enumerator service.
      • Case-insensitive.
        • You can specify both inclusion and exclusion rules; one set per line.
        • Use * at the end of the string for wild cards.

        • Use the "|" character to separate multiple entries.

        • Enter site root URLs only. DO NOT enter sub-site URLs.
          Example:
          include=https://[tenant].sharepoint.com/site1|https://[tenant].sharepoint.com/sites*
          exclude=https://[tenant].sharepoint.com/excluded

          NOTE: The site collections enumerator filters top-level site collections.

          The enumerator does NOT filter sub-sites.

  3. External Domains:
    1.  If the remote farm lives in an untrusted AD domain, specify the Fully Qualified Domain Name or IP address of the Domain Controller to use to properly process any AD security group used by the remote SharePoint farm.
      The information must be added in the following format:
         DomainController: <FQDN_or_IP>
      • Leave empty if the content source and the indexing farm are in the same domain
      • Multiple domains must be added on separate lines
      • Login information for each domain must be added in the Login section in the same order in which the domains are added in this section.
      • Optional parameters can be added for each domain, on the same line as the FQDN or IP. The parameters must be separated by semi-colons.
        • GroupsExcludedFromExpansion:
          • This is a comma (,) separated list of groups that will be excluded from expansion during Security Sync
        • EnableSSL:
          • If set to "True", LDAP queries are used to retrieve the group membership.
          • The port number (default: 636) must also be added to the Domain Controller FQDN or IP.



  4. Disable Security: Click this option if you want to ignore security and make all of your content public.
  5. Native Properties:
    1. If this flag is set, all the metadata and properties are returned directly as native SharePoint crawled properties.
    2. The Native Properties operation simulates the built-in SharePoint crawler and does not use the Connector metadata mapping page.
    3. For this reason, uncheck this box if you want to use the Metadata page.
  6. Compound Properties:
    1. Enter a list of properties that you want returned as a single appended string.
    2. If you enter more than one property, specify the delimiter using Compound Delimiters as explained below:
      1. For each of specified properties, the SharePoint 2013/16/19 Connector returns a property with a _COMPOUND suffix.
      2. The value of this property is a single appended string from multiple values, which is useful to display in your search results.
        1. For example, if the multi-value property SAMPLE has the string values SAMPLE and SAMPLE1, the SAMPLE_COMPOUND property takes the SAMPLE,SAMPLE1 string value.
  7. URL for SharePoint Site Collections Enumerator Service:
    1. Only applies if Automatically - Fetch Site Collection setting that you specify in step 11, is on.
    2. Enter a list of Site Enumerator services instances for each web application to enumerate.
    3. Make sure that the URL is formed with a site that exists in your collection.
      1. For example, if you have 2 sites in your collection:
        1. http://webapp/site1 and http://webapp/site2,
      2. Then your enumerator URL should be: 
        1. http://webapp/site1/_vti_bin/siteenumerator/sitesenumerator.svc 
          or
        2. http://webapp/site2/_vti_bin/siteenumerator/sitesenumerator.svc

  8. Ignore certificate errors: Check this option to ignore certificate errors.


  9. Honor "Do Not Index" setting:

    By default, this check box is selected. Leave the default setting unless you want to index all the found libraries.

    1. If you make a change to this setting, you must recrawl your content.

    2. The setting also applies to sites.

    3. When checked, lists belonging to sites which are marked with the No index flag are not crawled and the sites do not show up in the Mailboxes/Datastores list in Connector Framework.

      Note: This setting is not inherited by sub-sites and only works if you use the Auto-Fetch functionality for retrieving the site-collections from your SharePoint environment.

  10. Compound Delimiters:
    1. Specify a delimiter to separate values in your compound properties.
    2. For example, specify a comma (,) to use a comma- delimited string
  11. Automatically - Fetch Site Collections
    • OFF (optional): Default setting. When set to OFF, you must explicitly enter each site collection you wish to index.
    • ON: Use this drop-down setting to use the Site Collection Enumerator to collect all the site collections from the farm to index.


  12. Click Save.

Configure the Mailbox/Datastore Options

You must configure your mailboxes, or datastores, for the SharePoint 2013/16/19 Connector, using the following steps:

  1. Mailbox/Datastore Options: Click and see the Mailbox/Datastore Options page.
    1. Sync New Mailboxes/Datastores:
      1. If this flag is set, new Mailboxes/Datastores are automatically marked as active in the Mailboxes/Datastores list, and are processed during the next crawl.
      2. Otherwise, you must use the Mailboxes/Datastores list to activate the mailboxes and datastores automatically.
    2. Subtractive Sync:
      1. If this flag is set, Mailboxes/Datastores which are not found in Mailbox/Datastore load task are automatically removed from the Mailboxes/Datastores list and therefore from the index on the next incremental crawl.
      2. If you do not select this operation, use Mailboxes/Datastores list to delete, or to deactivate, the Mailboxes/Datastores manually.
  2. Site Title Filter: Filter the SharePoint site titles whose content you want to index.
    1. Each entry is case-insensitive.
    2. Use the wildcard * on either end of your entry, and use a | to separate your entries.
    3. For example, specify title1|*title2|*title3*.
  3. Site URL Filter:
    1.  Filter the SharePoint URLs whose content you want to index. Each entry is case-insensitive.
    2. Use the wildcard * at the end of your entry, and use a pipe character (|) to separate your entries.
    3. For example, specify:
       https://sitecollection.sharepoint.com/site1|https//sitecollection.sharepoint.com/sites*
  4. The Site URL Filter works like the Site collections/Site collections Enumerator Filter when you use the Site Collection Enumerator.
  5. Web Template Id Filter:
    1. Filter the SharePoint web template IDs whose content you want to index.
    2. Each entry is case-insensitive and must not include the number.
    3. Use the wildcard * at the end of your entry, and use a "|" character to separate your entries.
    4. For example, specify:
         MPS|STS|SRCHEN|SPS*


  6. Click Save.

Datastores/Datastore Types Job Frequency Requirements

  • After your connection is configured, you must run the datastore load job and datastore type load jobs in order to detect the SharePoint sites and metadata to index.

  • We recommend scheduling these jobs periodically once your content has been successfully indexed, so that new SharePoint sites and metadata can be properly detected and supported.

Content Configuration Specifics

For a given content source, you can apply additional filters to decide which list or item to crawl.

Use the following syntax for the filters:

  • LISTURL =url1,url2,...
  • LISTTITLE=Title1,Title2,...
  • FOLDERURL=url1,url2,...
  • RECORDSONLY=[metadata name][=/<>]['expected value'/NULL]
  1. Separate filter items using a comma (,) and separate filters with a character pipe (|) character.
  2. Filters support trailing wild cards.
  3. To use a literal comma (,) or a pipe character (|), precede the character with a backslash (\,), or (\|), respectively.
  4. If any changes are made to this field, a full crawl is required to properly refresh the index.
  5. The records only option can be used to only return metadata for the items matching the condition specified.
    1. No file or binary content is retrieved for such items.

  6. By default, connector returns SharePoint sites as search result items.
  7. To exclude these entries from search results, disable Site content type using the content types filter.

Extracting HTML Text and Metadata from Documents

You can extract HTML text and metadata from your source system.

  • This is optional.

Extracted information includes:

  • Available text and metadata which can be mapped to metadata properties. 
  • HTML metadata tags from HTML pages.
  • Available HTML tags and links for use as metadata from HTML documents

Other Configuration Specifics

If during incremental crawl your changes are not picked up by the connector, check the web.config file for the settings:

  • ChangeAction and ObjectChanged

  • If one of these settings is missing from the configuration file, the Connector checks for all changes.

  • To property map well known groups from SharePoint (like Everyone except external users, All Authenticated Users and authenticated Users), use the following script for Group load script:

Group Load Script
Copy 
dim sysId as string = HOST.GetSystemId()

If sysId = "true"HOST.SetAdId("NT AUTHORITY\Authenticated Users")
End If


If sysId = "windows"HOST.SetAdId("NT AUTHORITY\Authenticated Users")
End If


If sysId.startswith("spo-grid-all-users")
HOST.SetAdId("NT AUTHORITY\Everyone")
End If