Configuring LLM Connections for Agentic RAG

SmartHub supports agentic Retrieval-Augmented Generation (RAG) for conversational search. Agentic RAG enables SmartHub to orchestrate reasoning steps, call external tools, and produce more accurate responses by leveraging large language models (LLMs) as planning engines.

To enable these capabilities, SmartHub requires a connection to an LLM provider. This page lets administrators create and manage connections to OpenAI and Azure OpenAI.

To configure your LLM, do the following:

  1. In the SmartHub administration portal, under Conversational Search, click LLM Configuration.

  2. On the LLM Configurations page, click Add LLM to create a new LLM connection. Alternatively, you can edit an existing connection by clicking on the Connection Name.

  3. On the Large language Model modal window, provide a name for your LLM connection in the Connection Name field.

  4. In the Provider field, select OpenAI or Azure OpenAI from the drop-down menu.

    1. If you selected OpenAI, complete the following fields:

      1. In the Endpoint field, you can specify an OpenAI endpoint to send API requests. For example, api.openai.com. This field is optional.

      2. In the API Key, field, provide the API key for your OpenAI resource.

      3. In the Model ID field, enter the model ID of your OpenAI resource. For example, gpt-4o.

      4. In the Organization ID field, enter the organization ID of your OpenAI resource. This field is optional.

      5. In the Service ID field, enter the id of the service account for your OpenAI resource. This field is optional.

      6. In the Maximum Context Tokens field, enter the total token capacity the model can handle in a single prompt and response. This limit controls how much content the LLM can process at once, including the user query, RAG document context, instructions, reasoning steps, and the model’s output. A higher token limit supports better conversational memory and more accurate agentic RAG behavior, but also increases processing costs.

        By default, this value is 30000.

    2. If you selected Azure OpenAI, complete the following fields:

      1. In the Deployment URL field, enter the deployment or endpoint URL for your Azure OpenAI resource. For example, https://<your-resource-name>.openai.azure.com/.

      2. In the API Key field, enter the API key for your Azure OpenAI resource.

      3. In the Deployment Name field, enter the deployment name for your Azure OpenAI resource.

      4. In the Service ID field, enter the id of the service account for your Azure OpenAI resource. This field is optional.

      5. In the Model ID field, enter the model ID of your Azure OpenAI resource. For example, gpt-4.1.

      6. In the API Version field, enter the API version of your Azure OpenAI resource. for example, 2024-06-01.

      7. In the Maximum Context Tokens field, enter the total token capacity the model can handle in a single prompt and response. This limit controls how much content the LLM can process at once, including the user query, RAG document context, instructions, reasoning steps, and the model’s output. A higher token limit supports better conversational memory and more accurate agentic RAG behavior, but also increases processing costs.

        By default, this value is 30000.

  5. Click Save.

  6. Verify that the Status control is turned on for your LLM configuration.

  7. Additionally, you can click the Delete icon to remove your LLM configuration.