What is a Connector?

    Do I Buy or Build My Connector (video)?

Connectors, often called "indexing connectors," extract content from source systems and transmit it to a search engine, such as Azure Search, or Elasticsearch for indexing.

  • Each enterprise repository typically has a specific way to extract content (access method or API), a particular layout of content (schema), and specific security capabilities.
  • Therefore, each type of system may need a connector developed specifically for it.
  • A connector establishes a secure connection to the source system and maps the content, including metadata and attachments, from the source system schema to the search engine schema.
  • It then extracts content and feeds it to the search engine in a process called crawling.

There are two main types of crawls:

  • Full Crawls, which extract all desired content.
  • Incremental Crawls, which extract only content which has changed since the last crawl.

Connector Functions


Indexing Connectors

BA Insight provides a wide range of Indexing Connectors.

Each connector is developed and maintained for a particular source system.

There are two types of connectors:

  • SQL-based Connectors
    • For source systems that expose content via an underlying database.
    • These connectors use a common framework with template-based administrative screens.
    • The SQL calls are available for tailoring, either for performance optimization or to support advanced scenarios.

  • Web Service Connectors
    • For source systems that publish APIs for content access.
    • Web service connectors include a number of functions and communicate to the Connector Framework through a published Web Services-based API.
    • The structure of these connectors is shown below.

Web Service Connector Architecture

 

  • All connectors share a high throughput, light touch approach to selecting and extracting content.
  • They are all agentless: they do not require any software to be installed on the source system and can communicate over a network to remote systems.
  • They only require READ access, so there is no risk of compromising source systems.
  • Many of BA Insight’s Indexing Connectors also can act as Dataset Connectors.
    • For example, an SQL system may have an associated file system for raw storage, or a file-based system may have an associated database holding metadata.
    • In these cases, both the file and the metadata are indexed as a single item using an associated crawl.

If You Need to Create a New or Custom Connector

BA Insight has extensive experience in creating and maintaining indexing connectors, and a proven process for approaching new systems.

Connector Framework provides facilities for:

  • Testing
  • Troubleshooting
  • Optimizing content extraction, which makes creating new connectors faster and simpler.

Connectors built on this framework also inherit many powerful features such as Targets and Smart Mapping, and present a consistent and effective interface to administrators.

There are two main facilities for creating new connectors.

  • Universal SQL connector toolkit
    • Supports secure indexing for any SQL-based source system with the flexibility to tailor the way database content is composed and transformed into indexed items.

  • Web Services API
    • Developers can also use this API to integrate crawling into their system or to create new connectors themselves.
    • BA Insight also provides services to create custom connectors and/or mentor developers who wish to create connectors.

How to Install a Connector

Each connector has specific configuration steps. See the topic for your connector for details.

For a high-level procedure of installing any connector into BA Insight Connector Framework, see: