How to Configure the SharePoint Online Connector Using Web.config

The SharePoint Online Connector does not retrieve 'hidden' properties for the items crawled.

• If there are properties which you want to include from this category, they must be listed in connector's web.config file under "ListOfHiddenColumnsToInclude".
• A complete record of the 'hidden' properties which have been ignored is listed in the connector logs.
• Note, you must set your Log level to DEBUG.

AppSettings

CSOM and General Settings

1. <add key="CSOMTimeout" value="600000"/>: This specifies the CSOM timeout configuration.
   
2. <add key="UserAgent" value="ISV|BA Insight|Sharepoint Online Connector/1.0"/>: This specified the User Agent presented with each CSOM request.
   
3. <add key="MaxDegreesOfParallelism" value="16"/>: This is the value when parallelism is used to process data.  Adjust this value down if you encounter Throttling. If this setting is not present, the MaxDegreesOfParallelism will be 16.
4. <add key="ThrottlingRetryMode" value="True"/>: If SharePoint Online rejects a request due to Throttling, the request will be re-tried if this is set to True.
   
5. <add key="ThrottlingInitialTimeSeconds" value="30"/>: This is the amount of time to wait before retrying when throttling occurs.
   
6. <add key="ThrottlingRetries" value="5"/>: This is the number of retries when throttling occurs before the request will be logged as a failure.
   
7. <add key="MemoryCacheTimeLimitHours" value="24"/>: This is the time before caching expires. Caching will automatically be flushed between Security Sync Loads User and Group tables and executes the mapping based on the connector configuration. and each crawl.
   
8. <add key="HtmlCleanRegex" value="&lt;[a-zA-Z\/][^&gt;]*&gt;"/>: This is the regex for cleansing html content
9. <add key="EnableSecurityUnifiedMembersCaching" value="True"/>: This allows you to enable or disable memory caching for security sync for unified members.
10. <add key="EnableSecurityRoleMembersCaching" value="True"/>: This allows you to enable or disable memory caching for security sync for role members.
11. <add key="EnableSecuritySharePointMembersCaching" value="True"/>: This allows you to enable or disable memory caching for security sync for SharePoint members.
12. <add key="MemoryCacheThreshold" value="0.90"/>: This allows you to control the amount of memory caching that you want to allow for your connector. For example, if this is set to "0.90", this setting will disable the continued use of cache when Memory used exceeds 90% of the physical memory.
    

Metadata Retrieval Settings

1. <add key="PropertyNamesToExclude" value=""/>: This is a comma separated list of properties that you want to exclude from item metadata Provides context with details such as the source, type, owner, and relationships to other data sets. Metadata provides details around the item being crawled by Connectivity Hub..
   
2. <add key="ListOfHiddenColumnsToInclude" value=""/>: This is a comma separated list of hidden properties to be included in item metadata. For example, "Created_x0020_By,Modified_x0020_By,SharedWithUsers,SharedWithDetails"
   

Security and Azure Active Directory Settings

1. <add key="GroupLoadPaginated" value="True"/>: This determines if Groups should be loaded by page or single retrieval.
   
2. <add key="GroupPageSize" value="5000"/>: This specifies the page size that will be retrieved if GroupLoadPaginated="True"
   
3. <add key="EnableFullSecurityCache" value="True"/>: Caching is used upon Group Loading. If memory becomes excessive during Security Synchronization, you can disable this setting.
   
4. <add key="GraphAPITimeout" value="5"/>: This is the Microsoft Graph API timeout in seconds.
   
5. <add key="GetUsersPageSize" value="999"/>: This is the Microsoft Graph user retrieval page size. The maximum size for this field is 999.
   
6. <add key="GetGroupsPageSize" value="999"/>: This is the Microsoft Graph group retrieval page size. The maximum size for this field is 999.
   
7. <add key="GetGroupMembersPageSize" value="999"/>: This is the Microsoft Graph membership retrieval page size. The maximum size for this field is 999.
   
8. <add key="UserProperties" value=""/>: This is a comma separated list of the Azure Active Directory An identity and access management solution from Microsoft that helps organizations secure and manage identities for hybrid and multicloud environments. user properties that are retrieved during a security sync. By default, security will return "id", "userPrincipalName", "displayName", "accountEnabled","mail" =. To return additional fields, add them to the comma separated list. For example, "onPremisesSecurityIdentifier, employeeId, givenName, mail, onPremisesDistinguishedName, onPremisesDomainName, onPremisesUserPrincipalName, surname, otherMails, proxyAddresses"
   
9. <add key="GroupProperties" value=""/>: This is a comma separated list of the Azure Active Directory A directory service for Windows domain networks. A hierarchical structure that stores information about objects on the network. Used to manage permissions and control access to critical network resources. group properties that are retrieved during a security sync. By default, security will return "id", "displayName", "mail","groupTypes". To return additional fields, add them to the comma separated list. For example, "onPremisesSecurityIdentifier, mail, onPremisesNetBiosName, onPremisesSamAccountName".
   
10. <add key="SPOAuthorityEndpoint" value="https://login.microsoftonline.com"/>: This is the SharePoint Online authentication endpoint. You can change this value to fit your SharePoint Online instance.
    
11. <add key="GraphEntrypoint" value="https://graph.microsoft.com"/>: This is the Microsoft Search Graph API endpoint. You can change this value to fit your Microsoft Search Graph instance.

12. <add key="TenantDomain" value="onmicrosoft.com" />: This is the tenant domain. If the tenant domain is not "onmicrosoft.com" use this setting to change the domain.

13. <add key="TenantName" value""/>:This allows you to override your tenant name by specifying a vanity link for your SharePoint Online connector configuration