Salesforce Quota and Best Practices

Quota

Salesforce.com imposes an API Call limit to ensure equitable access to the hosted service for all subscribers.

Salesforce organizations are subject to a maximum number of API calls that can be made against Salesforce.com servers in any 24-hour period (which is specified by the license).

For more information, see Introducing SOAP API.

Quota Impact

The Salesforce Connector implements as many Salesforce recommendations as possible to minimize the number of API calls.

Every task (security synchronization, data store types load, crawling, and so on) leverages the API in the same way.

The factors which have the most impact on the API quota are:

• Complexity of entities indexed such as contacts, accounts, cases, and so on
• Number of records and the number of attachments/files associated with each indexed record
• Frequency at which Salesforce is checked for updates

Search Indexing Approach

The data to be indexed is gathered in batches at a maximum number of 1,000 items per API call (the number of items retrieved is controlled by Salesforce).

• If there are more records than the limit, additional API calls are made.

• For most entities (accounts, opportunities, contact, lead, Quote, users), all the metadata Provides context with details such as the source, type, owner, and relationships to other data sets. Metadata provides details around the item being crawled by Connectivity Hub. to be indexed can be collected at one time as part of the batch request.

• As a result, a single API call is required to fetch 1,000 records of those entities.

• In other words, each record will consume 1/1,000^th of an API call.

However, there are entities that require multiple calls, such as Cases. For Cases, the Salesforce Connector requires 1 API call per 1,000 cases to index, plus 1 API call per attachment for each case.

For example, if on average a case has 3 attachments, indexing 1 case will consume:

• 1/1000 for the case metadata + 3 x 1 call per attachment = 3.001 API calls per case.
• In other words, for 10,000 customer cases, you consume approximately 30,010 calls.

The records' metadata is retrieved by using a single API call during the item enumeration phase and any attachments that are queried for each attachment in the item definition phase, where you can find the list with all the objects that have a parent-to-child relationship with the Attachment object.

If a custom SOQL query is used, the behavior is similar to a single out-of-the-box entity.

The query result is treated as a data store entity (has the name SOQL in the repository list) and is processed accordingly.

BA Insight Recommendations for Large Salesforce Subscriptions Indexing

Recommendations

In order to properly index your Salesforce data, consider the impact of the initial full crawl and on the subsequent incremental crawls Scanning and capturing only new data from all of your content sources. This data did not exist when the last crawl was run. against you API quota.

There are two dimensions to scaling for a full crawl:

• How many crawl servers are deployed in your SharePoint farm?
• How much of your API quota can you can allocate to crawling your Salesforce data?

The more crawl servers you have, the faster you will consume your API quota.

• Run a test crawl to estimate the speed at which your environment executes a full crawl on your Salesforce content.

• This number is the approximate run rate at which your quota is consumed.

• Based on your run rate and on the total number of records to index (this information can be found using the Report functionality in Salesforce), you can estimate the API quota limit that you need to perform a full crawl using your full farm capabilities.

• If this estimated API quota limit is greater than your current API quota, the crawl will eventually exceed your quota, and at that point the Connector will start failing to retrieve records.

• The error will be logged and the crawl will go forward trying to fetch another record.

• You must tune your environment, to lower the impact on the API quota, and execute a new full crawl to recollect the missed items.

To ensure that you remain below your API quota, BA Insight recommends:

• The simplest approach is to purchase a quota increase from Salesforce.com for the initial full crawl so that the crawl completes at full speed while remaining within your quota limit.
• You can use the estimated API calls explained above.
• If you require a quota increase, contact Salesforce.
• Use the SharePoint Crawl Impact Rules to control how many items are crawled in parallel.
• By reducing the number of items crawled in parallel, you can lower the impact on your API quota (at the expense of a longer full crawl duration). With a lower run rate on the API quota, you are less likely to exceed your quota limit.
• Split your content to crawl into multiple content sources. For example, 1 instance per entity to index and scatter the crawls for these content sources over multiple 24-hour time periods.

Incremental Crawls

Incremental crawls have the same per-record footprint as full crawls.

However, since incremental crawls only re-crawl items that have changed since the previous crawl, the number of items reprocessed is significantly less.

A typical incremental crawl processes no more than 10% of the original corpus, where the average is closer to 2%.

However, each environment is different. Estimate this range based on your own data.