Chapter 1. Configure and deploy gateway policies

Procedure

1. Set the following environment variable by running the following command:

   $ export KUADRANT_GATEWAY_NS=api-gateway \
     export KUADRANT_GATEWAY_NAME=ingress-gateway \
     export KUADRANT_DEVELOPER_NS=toystore \
     export KUADRANT_AWS_ACCESS_KEY_ID=xxxx \
     export KUADRANT_AWS_SECRET_ACCESS_KEY=xxxx \
     export KUADRANT_ZONE_ROOT_DOMAIN=example.com \
     export KUADRANT_CLUSTER_ISSUER_NAME=self-signed

   • KUADRANT_GATEWAY_NS: Namespace for your gateway in OpenShift Container Platform.
   • KUADRANT_GATEWAY_NAME: Name of your gateway in OpenShift Container Platform.
   • KUADRANT_DEVELOPER_NS: Namespace for the example Toystore app in OpenShift Container Platform. You can replace this value with the name of the application you want to use.
   • KUADRANT_AWS_ACCESS_KEY_ID: DNS provider access key ID. In this example, AWS is used. You can replace this value with your DNS provider information.
   • KUADRANT_AWS_SECRET_ACCESS_KEY: DNS provider secret access key with permissions to manage your DNS zone. In this example, AWS is used. You can replace this value with your DNS provider information.
   • KUADRANT_ZONE_ROOT_DOMAIN: The root domain associated with your DNS zone ID. In this example, the OpenShift Container Platform secret containing the credentials for the DNS provider is AWS Route53. You can replace this value with your DNS provider information.
   • KUADRANT_CLUSTER_ISSUER_NAME: Name of the certificate authority or the issuer for TLS certificates.

2. Create the namespace for the application by running the following command:

   $ oc create ns ${KUADRANT_DEVELOPER_NS}

3. Deploy your application to the namespace you specified by running the following command:

   $ oc apply -f https://raw.githubusercontent.com/Kuadrant/Kuadrant-operator/main/examples/toystore/toystore.yaml -n ${KUADRANT_DEVELOPER_NS}

   You can replace the Toystore application with the path to the one you want to use.

Procedure

1. Create the namespace where you want your Gateway CR applied in by running the following command:

   $ oc create ns ${KUADRANT_GATEWAY_NS}

2. Create the Secret credentials in the same namespace as the Gateway CR namespace by running the following command:

   $ oc -n ${KUADRANT_GATEWAY_NS} create secret generic <aws_credentials> \
     --type=kuadrant.io/aws \
     --from-literal=AWS_ACCESS_KEY_ID=$KUADRANT_AWS_ACCESS_KEY_ID \
     --from-literal=AWS_SECRET_ACCESS_KEY=$KUADRANT_AWS_SECRET_ACCESS_KEY

   Replace <aws_credentials> with the secret you want to use.

Procedure

1. Create a Gateway custom resource (CR), such as {KUADRANT_GATEWAY_NAME}.yaml, that has the following information:

   Example Gateway CR

   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: ${KUADRANT_GATEWAY_NAME}
     namespace: ${KUADRANT_GATEWAY_NS}
     labels:
       kuadrant.io/gateway: "true"
   spec:
       gatewayClassName: openshift-default
       listeners:
       - allowedRoutes:
           namespaces:
             from: All
         hostname: "api.${KUADRANT_ZONE_ROOT_DOMAIN}"
         name: api
         port: 443
         protocol: HTTPS
         tls:
           certificateRefs:
           - group: ""
             kind: Secret
             name: api-${KUADRANT_GATEWAY_NAME}-tls
           mode: Terminate

   Important

   If you require cryptographic identity verification, see "Create a Gateway object for use with the X.509 format" for a Gateway object example to use instead of this default.

2. Apply the Gateway CR by running the following command:

   $ oc apply -f {KUADRANT_GATEWAY_NAME}.yaml

Verification

1. Check the status of your Gateway object by running the following command:

   $ oc get gateway ${KUADRANT_GATEWAY_NAME} -n ${KUADRANT_GATEWAY_NS} -o=jsonpath='{.status.conditions[?(@.type=="Accepted")].message}{"\n"}{.status.conditions[?(@.type=="Programmed")].message}'

   Example output

   Resource accepted
   Resource programmed, assigned to service(s) ${KUADRANT_GATEWAY_NAME}.${KUADRANT_GATEWAY_NS}.svc.cluster.local:443

2. Check the status of your HTTPS listener by running the following command:

   $ oc get gateway ${KUADRANT_GATEWAY_NAME} -n ${KUADRANT_GATEWAY_NS} -o=jsonpath='{.status.listeners[0].conditions[?(@.type=="Programmed")].message}'

   The HTTPS listener exists, but is not programmed or ready to accept traffic because you do not have valid certificates available.

Procedure

1. Create a TLSPolicy custom resource (CR) such as <toystore_tls.yaml>, that has the following information:

   Example TLSPolicy CR

   apiVersion: kuadrant.io/v1
   kind: TLSPolicy
   metadata:
     name: ${KUADRANT_GATEWAY_NAME}-tls
     namespace: ${KUADRANT_GATEWAY_NS}
   spec:
     targetRef:
       name: ${KUADRANT_GATEWAY_NAME}
       group: gateway.networking.k8s.io
       kind: Gateway
     issuerRef:
       group: cert-manager.io
       kind: ClusterIssuer
       name: ${KUADRANT_CLUSTER_ISSUER_NAME}

2. Apply your TLSPolicy CR by running the following command:

   $ oc apply -f <toystore_tls.yaml>

   Replace <toystore> with the name of your TLSPolicy object.

1. On an incoming request, the gateway evaluates the matching rules and predicates from the TokenRateLimitPolicy resources.
2. If the request matches, the gateway prepares the necessary rate limit descriptors and monitors the response.
3. After receiving the response, the gateway extracts the usage.total_tokens field from the JSON response body.
4. The gateway then sends a RateLimitRequest to the Limitador Operator (Limitador), including the actual token count as a hits_addend.
5. Limitador tracks the cumulative token usage and responds to the gateway with OK or OVER_LIMIT.

Procedure

1. Create a TokenRateLimitPolicy YAML, for example, tokenratelimitpolicy.yaml that includes the following information:

   Example TokenRateLimitPolicy YAML

   apiVersion: kuadrant.io/v1alpha1
   kind: TokenRateLimitPolicy
   metadata:
     name: llm-protection
   spec:
     targetRef:
       group: gateway.networking.k8s.io
       kind: Gateway
       name: ai-gateway
     limits:
       free-users:
         rates:
           - limit: 10000
             window: 24h
         when:
           - predicate: request.path == "/v1/chat/completions"
           - predicate: |
               auth.identity.groups.split(",").exists(g, g == "free")
         counters:
           - expression: auth.identity.userid
       pro-users:
         rates:
           - limit: 100000
             window: 24h
         when:
           - predicate: request.path == "/v1/chat/completions"
           - predicate: |
               auth.identity.groups.split(",").exists(g, g == "pro")
         counters:
           - expression: auth.identity.userid

   • Choose a filename for the policy that makes sense in your environment.
   • The spec.limits.free-users.rates.limit parameter is set for 10,000 tokens per day for free tier.
   • The spec.limits.free-users.when.predicate parameter is set to inference traffic only.
   • The spec.limits.pro-users.rates.limit parameter is set for 100,000 per day for the pro user.
   • The spec.limits.pro-users.when.predicate parameter is set to inference traffic only.
   • The predicates must match each other in each section.

2. Apply the TokenRateLimitPolicy policy by running the following command:

   $ oc apply -f <tokenratelimitpolicy.yaml> -n <gateway_namespace>

   • Replace <tokenratelimitpolicy.yaml> with the filename you used.
   • Replace <gateway_namespace> with your gateway namespace.

3. Check the status of the policy to ensure it was accepted and enforced on the target Gateway object. Look for conditions with type: Accepted and type: Enforced with status: "True".

   $ oc get <tokenratelimitpolicy.yaml> llm-protection -n <gateway_namespace> -o jsonpath='{.status.conditions}'

   Replace <tokenratelimitpolicy.yaml> with the filename you used. Replace <gateway_namespace> with your Gateway object namespace.

4. Send requests to your API endpoint, including the required authentication details, by running the following command:

   $ curl -H "Authorization: <auth_policy>" \
        -d '{"model": "gpt-4", "messages": [{"role": "user", "content": "Hello"}]}' \
        <api_endpoint>

   Replace <auth_policy> and <api_endpoint> with your values.