1. Documentation
×

Configuring and deploying OpenShift Lightspeed

After the OpenShift Lightspeed Operator is installed, configuring and deploying consists of three tasks. First, you create a credential secret using the credentials for your Large Language Model (LLM) provider. Next, you create the OLSConfig Custom Resource (CR) file that the Operator uses to deploy the service. Finally, you verify that the Lightspeed service is operating.

The instructions assume that you are installing OpenShift Lightspeed using the kubeadmin user account. If you are using a regular user account with cluster-admin privileges, read the section of the documentation that discusses RBAC.

Creating the credentials secret using the web console

Create a file that is associated with the secret key used to access the API of your LLM provider. You can use API tokens to authenticate your LLM provider. Additionally, Microsoft Azure supports authentication using Entra ID.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role. Alternatively, you are logged in to a user account that has permission to create a secret to store the Provider tokens.

  • You have installed the OpenShift Lightspeed Operator.

Procedure

  1. Click the plus button in the upper-right corner of the OpenShift web console.

  2. Paste the following YAML content into the text area:

    Example credential secret for LLM provider
    apiVersion: v1
kind: Secret
metadata:
  name: credentials
  namespace: openshift-lightspeed
type: Opaque
stringData:
  apitoken: <token_string_version> (1)
    1 The apitoken is not base64 encoded.

    The YAML element is always apitoken regardless of what the LLM provider calls the access details.
    Example credential secret for Microsoft Azure OpenAI
    apiVersion: v1
data:
  client_id: <base64_encoded_client_id>
  client_secret: <base64_encoded_client_secret>
  tenant_id: <base64_encoded_tenant_id>
kind: Secret
metadata:
  name: azure-api-keys
  namespace: openshift-lightspeed
type: Opaque

  3. Click Create.

Creating the credentials secret using the command line

Create a file that is associated with the secret key used to access the API of your LLM provider. You can use API tokens to authenticate your LLM provider. Additionally, Microsoft Azure supports authentication using Entra ID.

Prerequisites

  • You have access to the OpenShift CLI (oc) as a user with the cluster-admin role. Alternatively, you are logged in to a user account that has permission to create a secret to store the Provider tokens.

  • You have installed the OpenShift Lightspeed Operator.

Procedure

  1. Create a file that contains the following YAML content:

    Example credential secret for LLM provider
    apiVersion: v1
kind: Secret
metadata:
  name: credentials
  namespace: openshift-lightspeed
type: Opaque
stringData:
  apitoken: <token_string_version> (1)
    1 The apitoken is not base64 encoded.
    Example credential secret for Microsoft Azure OpenAI
    apiVersion: v1
data:
  client_id: <base64_encoded_client_id>
  client_secret: <base64_encoded_client_secret>
  tenant_id: <base64_encoded_tenant_id>
kind: Secret
metadata:
  name: azure-api-keys
  namespace: openshift-lightspeed
type: Opaque

  2. Run the following command to create the secret:

    $ oc create -f /path/to/secret.yaml

Creating the Lightspeed custom resource file using the web console

The Custom Resource (CR) file contains information that the Operator uses to deploy OpenShift Lightspeed. The specific content of the CR file is unique for each LLM provider. Choose the configuration file that matches your LLM provider.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role. Alternatively, you are logged in to a user account that has permission to create a cluster-scoped custom resource file.

  • You have installed the OpenShift Lightspeed Operator.

Procedure

  1. Click the plus button in the upper-right corner of the OpenShift web console.

  2. Paste the YAML content for the LLM provider you use into the text area of the web console:

    Example OpenAI custom resource file
    apiVersion: ols.openshift.io/v1alpha1
kind: OLSConfig
metadata:
  name: cluster
spec:
  llm:
    providers:
      - name: myOpenai
        type: openai
        credentialsSecretRef:
          name: credentials
        url: "https://api.openai.com/v1"
        models:
          - name: gpt-3.5-turbo
  ols:
    defaultModel: gpt-3.5-turbo
    defaultProvider: myOpenai
    Example Microsoft Azure OpenAI custom resource file
    apiVersion: ols.openshift.io/v1alpha1
kind: OLSConfig
metadata:
  name: cluster
spec:
  llm:
    providers:
      - credentialsSecretRef:
          name: credentials
        deploymentName: <azure_ai_deployment_name>
        models:
          - name: gpt-35-turbo-16k
        name: myAzure
        type: azure_openai
        url: <azure_ai_deployment_url>
  ols:
    defaultModel: gpt-35-turbo-16k
    defaultProvider: myAzure
    Example IBM WatsonX custom resource file
    apiVersion: ols.openshift.io/v1alpha1
kind: OLSConfig
metadata:
  name: cluster
spec:
  llm:
    providers:
      - name: myWatsonx
        type: watsonx
        credentialsSecretRef:
          name: credentials
        url: <ibm_watsonx_deployment_name>
        projectId: <ibm_watsonx_project_id>
        models:
          - name: ibm/granite-13b-chat-v2
  ols:
    defaultModel: ibm/granite-13b-chat-v2
    defaultProvider: myWatsonx

  3. Click Create.

Creating the Lightspeed custom resource file using the CLI

The Custom Resource (CR) file contains information that the Operator uses to deploy OpenShift Lightspeed. The specific content of the CR file is unique for each LLM provider. Choose the configuration file that matches your LLM provider.

Prerequisites

  • You have access to the OpenShift CLI (oc) and are logged in as a user with the cluster-admin role. Alternatively, you are logged in to a user account that has permission to create a cluster-scoped custom resource file.

  • You have installed the OpenShift Lightspeed Operator.

Procedure

  1. Create an OLSConfig file that contains the YAML content for the LLM provider you use:

    Example OpenAI custom resource file
    apiVersion: ols.openshift.io/v1alpha1
kind: OLSConfig
metadata:
  name: cluster
spec:
  llm:
    providers:
      - name: myOpenai
        type: openai
        credentialsSecretRef:
          name: credentials
        url: "https://api.openai.com/v1"
        models:
          - name: gpt-3.5-turbo
  ols:
    defaultModel: gpt-3.5-turbo
    defaultProvider: myOpenai

    For OpenShift AI vLLM use the same configuration as OpenAI but update the URL to point to your Virtual Large Language Model (vLLM) endpoint. If OpenShift Lightspeed operates in the same cluster as the vLLM model serving instance, you can point to the internal OpenShift service hostname instead of exposing vLLM with a route.
    Example Microsoft Azure OpenAI custom resource file
    apiVersion: ols.openshift.io/v1alpha1
kind: OLSConfig
metadata:
  name: cluster
spec:
  llm:
    providers:
      - credentialsSecretRef:
          name: credentials
        deploymentName: <azure_ai_deployment_name>
        models:
          - name: gpt-35-turbo-16k
        name: myAzure
        type: azure_openai
        url: <azure_ai_deployment_url>
  ols:
    defaultModel: gpt-35-turbo-16k
    defaultProvider: myAzure
    logLevel: DEBUG
    Example IBM WatsonX custom resource file
    apiVersion: ols.openshift.io/v1alpha1
kind: OLSConfig
metadata:
  name: cluster
spec:
  llm:
    providers:
      - name: myWatsonx
        type: watsonx
        credentialsSecretRef:
          name: credentials
        url: <ibm_watsonx_deployment_name>
        projectId: <ibm_watsonx_project_id>
        models:
          - name: ibm/granite-13b-chat-v2
  ols:
    defaultModel: ibm/granite-13b-chat-v2
    defaultProvider: myWatsonx
    logLevel: DEBUG

  2. Run the following command:

    $ oc create -f /path/to/config-cr.yaml

    The Operator deploys OpenShift Lightspeed using the information in YAML configuration file.

Verifying the OpenShift Lightspeed deployment

After the OpenShift Lightspeed service is deployed, verify that it is operating.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role.

  • You have access to the OpenShift CLI (oc).

  • You have installed the OpenShift Lightspeed Operator.

  • You have created the credentials secret and the OLSConfig Custom Resource configuration file.

Procedure

  1. In the OpenShift Container Platform web console, select the Developer perspective from the drop-down list at the top of the pane.

  2. Click the Project drop-down lsit.

  3. Enable the toggle switch to show default projects.

  4. Select openshift-lightspeed from the list.

  5. Click Topology.

    When the circle around the Lightspeed icon turns dark blue, the service is ready.

  6. Verify that the OpenShift Lightspeed is ready by running the following command:

    $ oc logs deployment/lightspeed-app-server -c lightspeed-service-api -n openshift-lightspeed | grep Uvicorn
    Example output
    INFO: 	Uvicorn running on https://0.0.0.0:8443 (Press CTRL+C to quit)

About Lightspeed and Role Based Access Control (RBAC)

Role-Based Access Control (RBAC) is a system security approach to restricting system access to authorized users who have defined roles and permissions.

OpenShift Lightspeed RBAC is binary. By default, not all cluster users have access to the OpenShift Lightspeed interface. Access must be granted by a user who can grant permissions. All users of an OpenShift cluster with OpenShift Lightspeed installed can see the Lightspeed button; however, only users with permissions can submit questions to Lightspeed.

If you want to evaluate the RBAC features of OpenShift Lightspeed, your cluster will need users other than the kubeadmin account. The kubeadmin account always has access to OpenShift Lightspeed.

Granting access to an individual user

This procedure explains how to grant access to an individual user.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role. Alternatively, you are logged in as a user with the ability to grant permissions.

  • You have deployed the OpenShift Lightspeed service.

  • You have access to the OpenShift CLI (oc).

Procedure

  1. Run the following command at the command line:

    $ oc adm policy add-cluster-role-to-user \
lightspeed-operator-query-access <user_name>

Granting access to a user group

This procedure explains how to grant access to a user group. If your cluster has more advanced identity management configured, including user groups, you can grant all users of a specific group access to the OpenShift Lightspeed service.

Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role. Alternatively, you are logged in as a user with the ability to grant permissions.

  • You have deployed the OpenShift Lightspeed service.

  • You have access to the OpenShift CLI (oc).

Procedure

  1. Run the following command at the command line:

    $ oc adm policy add-cluster-role-to-group \
lightspeed-operator-query-access <group_name>

Filtering and redacting information

You can configure OpenShift Lightspeed to filter or redact information from being sent to the LLM provider. The following example shows how to modify the OLSConfig file to redact IP addresses.

You should test your regular expressions against sample data to confirm that they are catching the information you want to filter or redact, and that they are not accidentally catching information you do not want to filter or redact. There are several third-party websites that you can use to test your regular expressions. When using third-party sites, you should practice caution with regards to sharing your private data. Alternatively, you can test the regular expressions locally using Python. In Python, it is possible to design very computationally-expensive regular expressions. Using several complex expressions as query filters can adversely impact the performance of OpenShift Lightspeed.
Prerequisites

  • You are logged in to the OpenShift Container Platform web console as a user with the cluster-admin role.

  • You have access to the OpenShift CLI (oc).

  • You have installed the OpenShift Lightspeed Operator and deployed the OpenShift Lightspeed service.

Procedure

  1. Modify the OLSConfig file and create an entry for each regular expression to filter. The following example redacts IP addresses:

    Example custom resource file
    spec:
  ols:
    queryFilters:
      - name: ip-address
        pattern: '((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.?\b){4}'
        replaceWith: <IP_ADDRESS>

  2. Run the following command to apply the modified OpenShift Lightspeed custom configuration:

    $ oc apply -f OLSConfig.yaml