SupportConsoleDevelopersStart a trialContact

Chapter 6. Audit logging and tracing

Configure compliance audit logging for Red Hat OpenShift Lightspeed to produce a durable, reconstructable record of AI actions. Audit logging supports regulatory requirements such as the EU AI Act by capturing who asked what, what the AI proposed, who approved it, and what the system executed. Audit logging is enabled by default on both OpenShift Lightspeed and agentic OpenShift Lightspeed.

6.1. About compliance audit logging
Copy link

OpenShift Lightspeed provides compliance audit logging to create a durable, reconstructable record of all AI actions across the system. This audit trail supports regulatory requirements such as the EU AI Act. The system captures every significant AI action: who asked what, what the AI proposed, who approved it, and what the system executed.

Audit logging is enabled by default on both OpenShift Lightspeed and agentic OpenShift Lightspeed. You configure audit behavior independently on each component through the spec.audit field of the OLSConfig and AgenticOLSConfig custom resources (CRs).

Dual-emission design

The system emits every audit event in two formats simultaneously:

  • Structured JSON to stdout — Always active. Your cluster log aggregation stack collects these entries from pod logs. JSON output is available even when no OpenTelemetry endpoint is configured.
  • OpenTelemetry (OTEL) spans — Active when you configure an OTEL collector endpoint. When no endpoint is configured, the system uses a no-op exporter and continues operating normally.

This dual-emission design ensures that audit records are available regardless of whether an OTEL backend is configured.

Trace identifier

Every audit event carries a trace_id field that links all log lines and OTEL spans belonging to the same interaction:

  • For OpenShift Lightspeed Q&A and troubleshooting interactions: the trace_id is derived from the conversation_id with hyphens removed.
  • For agentic OpenShift Lightspeed operations: the trace_id is derived from the AgenticRun resource metadata.uid with hyphens removed.

The trace_id doubles as the OTEL trace ID, so you can correlate structured log output with distributed traces in your observability platform.

OpenTelemetry GenAI semantic conventions
OpenShift Lightspeed telemetry—traces, metrics, and structured logs—aligns with the OpenTelemetry GenAI semantic conventions, version 1.41. This alignment applies to both the agentic system, which comprises the agentic operator and sandbox, and the OpenShift Lightspeed service (lightspeed-service). Standardized telemetry lets you interpret OpenShift Lightspeed traces, metrics, and logs with common OpenTelemetry tooling.

Traces

  • The agentic system emits a separate trace for each phase of an operation, rather than a single trace for the entire operation lifecycle. The OpenShift Lightspeed service emits a separate trace for each request.
  • Spans follow GenAI naming conventions, such as chat <model> for a model interaction and execute_tool <name> for a tool call, and carry gen_ai.* attributes.
  • Span kinds reflect the type of work: CLIENT for calls to the large language model (LLM) and INTERNAL for agent framework operations.
  • OpenShift Lightspeed distinguishes the requested model from the responding model through the gen_ai.request.model and gen_ai.response.model attributes.
  • The execute_tool <name> span represents any tool call, not only Model Context Protocol (MCP) tools. MCP semantic convention attributes are added only when the tool is MCP-sourced.

Metrics

The system emits gen_ai.* Prometheus histogram metrics. For the agentic sandbox, these are the primary metrics. For the OpenShift Lightspeed service, they are emitted alongside the existing ols_* metrics.

Structured logs

Structured logs use the OTEL JSON format and are written to both stdout and the OTEL log exporter, which can use PostgreSQL as a backend. Each record is emitted through a single path, known as the single-emission rule, to avoid duplicate log entries.

Deferred capabilities

The following capabilities are not yet part of this alignment:

  • Content capture controls
  • Evaluation events

  • Cache token attributes

    Custom resource serialization
    When an agentic operation completes, the agentic operator serializes the AgenticRun custom resource as a compliance record. The serialization captures the full .spec and the following metadata fields: name, namespace, creationTimestamp, and uid.

This serialized record is the authoritative compliance artifact for a given autonomous operation. Pod logs from the agentic sandbox and the OpenShift Lightspeed service supplement this record as forensic evidence.

Approver identity injection
OpenShift Lightspeed injects the approver’s identity into AgenticRun resources automatically, regardless of the approval path used. A mutating webhook intercepts all PATCH requests to AgenticRun resources and adds the identity of the authenticated user who submitted the approval. This applies to approvals made through the OpenShift Lightspeed console, kubectl, or the API directly.
Audit logging scope
Audit events are captured only in the backend components. The OpenShift Lightspeed console and the agentic console do not emit audit events. All user interactions that affect the audit record are captured by the backend services at the point of receipt and processing.

OpenShift Lightspeed integrates with the OpenTelemetry (OTEL) Collector to export audit and distributed trace data to your observability platform. This integration is one half of the dual-emission audit architecture; structured JSON output to stdout is the other half and is always active.

OTEL spans
Each audit event generates an OTEL span. The span carries the same fields as the corresponding JSON log entry and uses the interaction trace_id as the OTEL trace ID. This lets you view a complete interaction timeline as a distributed trace in any OTEL-compatible backend such as Jaeger or Tempo.
Endpoint configuration
You configure the OTEL Collector endpoint through the spec.audit.otel.endpoint field on the OLSConfig or AgenticOLSConfig CR. The endpoint must be a gRPC OTLP address, for example: 
spec:
  audit:
    otel:
      endpoint: grpc://otel-collector.observability.svc.cluster.local:4317

You can configure the OTEL endpoint on OLSConfig, AgenticOLSConfig, or both independently.

Graceful degradation

Configuring an OTEL endpoint is optional. When no endpoint is configured:

  • The system initializes a no-op OTEL exporter.
  • All audit events are still written as structured JSON to stdout.
  • The system does not generate errors or warnings due to a missing OTEL endpoint.

You can add or update the OTEL endpoint at any time by editing the CR. The operator reconciles the change without restarting the service.

6.3. Audit log event types
Copy link

Reference information for the audit events emitted by OpenShift Lightspeed and agentic OpenShift Lightspeed when audit logging is enabled.

6.3.1. Common fields
Copy link

Every audit log entry contains the following fields, regardless of the event type or component that emits it:

FieldTypeDescription

event

string

Event type identifier. See the event tables in this topic for valid values.

trace_id

string

Correlation identifier for the interaction. For OpenShift Lightspeed: derived from the conversation_id with hyphens removed. For agentic OpenShift Lightspeed: derived from the AgenticRun metadata.uid with hyphens removed. Doubles as the OTEL trace ID.

timestamp

string (ISO 8601)

Time at which the event occurred, in UTC.

user_id

string

Identifier of the user who initiated the interaction.

component

string

Name of the component that emitted the event, for example lightspeed-service or lightspeed-agentic-operator.

6.3.2. OpenShift Lightspeed events
Copy link

The lightspeed-service component emits a separate trace for each Q&A and troubleshooting request. Each trace contains the following spans:

SpanDescription

request.lifecycle

The overall request lifecycle. This is the root span for the request.

request.auth

Authentication and authorization of the request.

request.rag

Retrieval-augmented generation (RAG) context retrieval.

request.history

Conversation history retrieval.

chat <model>

Interaction with the large language model (LLM). The span name uses the requested model from the gen_ai.request.model attribute. The responding model is recorded separately in the gen_ai.response.model attribute.

execute_tool <tool-name>

A tool call. The tool is named in the span. Model Context Protocol (MCP) semantic convention attributes are added only when the tool is MCP-sourced.

request.store

Persistence of the request and response.

The following span events are emitted within these spans:

Span eventDescription

request.started

The request was received. Marks the start of an auditable interaction.

request.completed

The request completed. Marks the end of the interaction.

request.failed

The request failed.

rag.retrieved

RAG context was retrieved.

history.retrieved

Conversation history was retrieved.

gen_ai.choice

Completion or reasoning content returned by the model. When content capture is disabled, this event can be emitted without the completion or reasoning content attributes.

tool.call

A tool call was initiated.

tool.result

A tool call returned a result.

approval.requested

A human approval was requested.

approval.decision

An approval decision was recorded.

6.3.3. Agentic OpenShift Lightspeed events
Copy link

Agentic OpenShift Lightspeed emits audit events in two forms, as described by the dual-emission design: structured audit-log entries and OpenTelemetry (OTEL) span events. The two forms use different names for the same underlying event.

6.3.3.1. Agentic OTEL telemetry
Copy link

The agentic system emits a separate trace for each phase of an operation. The lightspeed-agentic-operator component emits phase span events, and the lightspeed-agentic-sandbox component emits telemetry for the model interactions and tool calls that run within a phase. The following table lists the telemetry emitted in each phase:

PhaseOperator span eventsSandbox telemetry

Analysis

  • agenticrun.received
  • sandbox.sa.created
  • sandbox.configmap.created
  • sandbox.rbac.created
  • sandbox.pod.created
  • agenticrun.analysis.completed
  • chat <model>
  • execute_tool <tool-name>
  • gen_ai.choice

Approval

agenticrun.approval.completed

None

Execution

  • sandbox.sa.created
  • sandbox.configmap.created
  • sandbox.rbac.created
  • sandbox.pod.created
  • agenticrun.execution.completed
  • chat <model>
  • execute_tool <tool-name>
  • gen_ai.choice

Verification

  • sandbox.sa.created
  • sandbox.configmap.created
  • sandbox.rbac.created
  • sandbox.pod.created
  • agenticrun.verification.completed
  • chat <model>
  • execute_tool <tool-name>
  • gen_ai.choice

Escalation

  • sandbox.sa.created
  • sandbox.configmap.created
  • sandbox.rbac.created
  • sandbox.pod.created
  • agenticrun.escalation.completed
  • chat <model>
  • execute_tool <tool-name>
  • gen_ai.choice

Terminal

agenticrun.terminal

None

In the operator column, agenticrun.* and sandbox.* are span events attached to the phase span, where the sandbox.* events record the operator creating the sandbox resources for the phase. In the sandbox column, chat <model> and execute_tool <tool-name> are spans, and gen_ai.choice is a span event that carries the model completion or reasoning content. When content capture is disabled, the gen_ai.choice event can be emitted without those content attributes.

6.3.3.2. Structured audit-log events
Copy link

Structured audit-log entries use a distinct set of event names that do not map directly to the span-event names. Examples include audit.agenticrun.received, audit.approval.received, and audit.agenticrun.terminal.

6.3.4. Additional fields for approval events
Copy link

Approval events include additional fields that identify the approval. The fields differ between the agentic system and the OpenShift Lightspeed service.

Agentic approval events, such as the structured audit.approval.received event and the agenticrun.approval.completed span event, include the following additional fields:

FieldTypeDescription

approver.uid

string

Unique identifier of the user who approved or rejected the proposal, as injected by the mutating webhook on the agentic-run PATCH request. This field is present regardless of the approval path used: console, kubectl, or API.

approver.username

string

Username of the user who approved or rejected the proposal, as injected by the mutating webhook on the agentic-run PATCH request. This field is present regardless of the approval path used: console, kubectl, or API.

Approval events emitted by the lightspeed-service component include the following additional field:

FieldTypeDescription

approval_id

string

Identifier of the approval associated with the event.

6.4. Audit logging configuration reference
Copy link

Reference information for the spec.audit fields on the OLSConfig and AgenticOLSConfig custom resources.

Both resources expose an identical spec.audit structure. You configure them independently: enabling or disabling audit logging on one does not affect the other.

Important

Audit logging is enabled by default. When the spec.audit field is absent from the CR, audit logging runs as if enabled: true is set. To turn off audit logging, you must explicitly set enabled: false.

6.4.1. spec.audit
Copy link

FieldTypeDefaultDescription

enabled

boolean

true

Controls whether audit logging is active. When absent, audit logging is enabled. Set to false to disable. There is no verbosity dial; audit logging is always full fidelity when enabled.

otel

object

—

Optional. OpenTelemetry Collector configuration. When absent, audit events are written as structured JSON to stdout only.

6.4.2. spec.audit.otel
Copy link

FieldTypeDefaultDescription

endpoint

string

—

gRPC OTLP endpoint for the OpenTelemetry Collector, for example grpc://otel-collector.observability.svc.cluster.local:4317. When absent, the system uses a no-op OTEL exporter and emits all audit events to stdout only.

The following example enables audit logging and configures an OTEL Collector endpoint on the OLSConfig CR: 

apiVersion: ols.openshift.io/v1alpha1
kind: OLSConfig
metadata:
  name: cluster
spec:
  audit:
    enabled: true
    otel:
      endpoint: grpc://otel-collector.observability.svc.cluster.local:4317
  llm:
    ...

The following example shows the equivalent configuration on the AgenticOLSConfig CR: 

apiVersion: ols.openshift.io/v1alpha1
kind: AgenticOLSConfig
metadata:
  name: cluster
spec:
  audit:
    enabled: true
    otel:
      endpoint: grpc://otel-collector.observability.svc.cluster.local:4317

6.4.4. Example: disable audit logging
Copy link

The following example disables audit logging on the OLSConfig CR: 

apiVersion: ols.openshift.io/v1alpha1
kind: OLSConfig
metadata:
  name: cluster
spec:
  audit:
    enabled: false
  llm:
    ...

6.5. Configure audit logging and tracing
Copy link

Configure compliance audit logging for Red Hat OpenShift Lightspeed to produce a durable, reconstructable record of AI actions. This feature supports regulatory requirements such as the EU AI Act by capturing who asked what, what the AI proposed, who approved it, and what the system executed. It is enabled by default on both OpenShift Lightspeed and agentic OpenShift Lightspeed.

Prerequisites

  • You have logged in to the OpenShift Container Platform web console as a user with the cluster-admin role.
  • You have access to the OpenShift CLI (oc).
  • You have installed the OpenShift Lightspeed Operator and deployed the OpenShift Lightspeed service.

Procedure

  1. To disable audit logging for OpenShift Lightspeed, set spec.audit.enabled to false in the OLSConfig CR: 

    spec:
  audit:
    enabled: false
    Important

    Disabling audit logging might violate regulatory compliance requirements. Ensure that disabling audit logs aligns with your organization’s compliance and security policies before proceeding.

  2. To disable audit logging for agentic OpenShift Lightspeed, set spec.audit.enabled to false in the AgenticOLSConfig CR: 

    spec:
  audit:
    enabled: false

  3. Optional: To export audit events to an OpenTelemetry Collector, add the spec.audit.otel.endpoint field to the relevant CR. You can configure this on OLSConfig, AgenticOLSConfig, or both, independently: 

    spec:
  audit:
    enabled: true
    otel:
      endpoint: grpc://otel-collector.observability.svc.cluster.local:4317

  4. Apply the updated CR: 

    $ oc apply -f <cr-file>.yaml

    Example output: 

    olsconfig.ols.openshift.io/cluster configured

Verification

  • To verify that audit events are being emitted for OpenShift Lightspeed, inspect the service pod logs and filter for JSON audit entries: 

    $ oc logs -n openshift-lightspeed deployment/lightspeed-app-server | grep '"event"'

    Each audit entry appears as a structured JSON line containing an event field, a trace_id, a user_id, and a timestamp.

  • To verify audit events for agentic OpenShift Lightspeed, inspect the agentic operator logs: 

    $ oc logs -n openshift-lightspeed deployment/lightspeed-agentic-operator | grep '"event"'
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top