Chapter 2. Observe MCP gateway connections
You can use observability in MCP gateway to debug silent failures, track destructive annotations and latencies, and configure an audit trail.
2.1. Enable Red Hat build of OpenTelemetry for the MCP gateway Copy linkLink copied to clipboard!
The MCP gateway uses Red Hat build of OpenTelemetry throughout all components to give you consistent logging, distributed tracing, and metrics. By using observability, you can achieve the following goals:
- Discover which component generated an error, such as Envoy, an MCP gateway component, or a backend MCP server.
- Understand how requests flow through the system components.
- See which tools were called and which MCP servers executed those tools.
- Understand where to find relevant logs for each component.
- Examine metrics about tool call patterns, success rates, and error rates.
- Examine distributed tracing across the request path.
Prerequisites
- You installed MCP gateway.
- You installed Connectivity Link.
-
You configured a
Gatewayobject. -
You are logged into a running OpenShift Container Platform cluster with an
adminrole. - You installed Red Hat build of OpenTelemetry.
- You configured the Red Hat OpenShift Distributed Tracing Platform for traces.
- You installed the OpenShift Logging Operator (Loki).
- You configured an S3-compatible persistent storage volume for Loki to store long-term MCP tool call logs.
Procedure
Use the following example YAML to create a collector in your MCP gateway deployment namespace:
Example OpenTelemetryCollector custom resource (CR)
apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: <mcp_otel_collector> namespace: <mcp_gateway_system> spec: mode: deployment config: receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 grpc: endpoint: 0.0.0.0:4317 exporters: otlp/tempo: endpoint: tempo-gateway-http.openshift-tracing.svc.cluster.local:4317 tls: insecure: true debug: verbosity: basic service: pipelines: traces: receivers: [otlp] exporters: [otlp/tempo, debug] logs: receivers: [otlp] exporters: [debug]-
Replace the value of
metadata.name:with the name you want to assign to this collector. -
Replace the value of
metadata.namespace:with the namespace of your MCP gateway deployment. -
The
spec.config.exporters:field value points to distributed tracing. -
The setting,
spec.config.exporters.otlp/tempo.tls.insecure: trueis for internal cluster communication without TLS.
-
Replace the value of
Apply the following environment variables to your OpenShift Container Platform cluster by running the following command:
$ oc set env deployment/mcp-gateway \ OTEL_EXPORTER_OTLP_ENDPOINT="http://<mcp_otel_collector>-collector.<mcp_gateway_system>.svc.cluster.local:4318" \ OTEL_EXPORTER_OTLP_INSECURE="true" \ OTEL_SERVICE_NAME="mcp-gateway"-
Replace
<mcp_otel_collector>with the name you used in yourOpenTelemetryCollectorCR. When Red Hat build of OpenTelemetry creates a collector, the service name is[metadata.name]-collector. -
Replace
<mcp_gateway_system>with the namespace of your MCP gateway deployment.
-
Replace
Optional. If you want to send traces to one collector and logs to a different one, set the following additional environment variables:
$ oc set env deployment/mcp-gateway \ OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://trace-collector.tracing-namespace.svc:4317" \ OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="http://log-collector.logging-namespace.svc:4317"-
Set
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTto override the endpoint for traces. -
Set
OTEL_EXPORTER_OTLP_LOGS_ENDPOINTto override the endpoint for logs.
-
Set
-
If you are using Red Hat OpenShift Distributed Tracing Platform, select your stack name as the data source in the OpenShift Web Console Observe > Traces dashboard. For example,
tempo-mcp. After you select the stack name, you can consult an attribute in your OpenShift Web Console OpenShift Observe > Traces dashboard to find where errors have occurred or to gather information for trend identification.
-
You can also filter by
service.name="mcp-gateway".
-
You can also filter by
-
When using Loki, use the Observe > Logs view and toggle the Structured view to filter by
mcp.session.id. -
When using Loki, use the Trace timeline view, look for the
Logicon next tospansto jump directly to the relevant logs.
Troubleshooting
-
If you do not see any traces, check if the
NetworkPolicyCR in your namespace allows traffic from the MCP gateway to the Red Hat build of OpenTelemetry collector service.
2.1.1. MCP gateway router spans for observability Copy linkLink copied to clipboard!
When enabled, the Model Context Protocol (MCP) router, ext_proc, emits trace spans for every request and can export structured logs through Red Hat build of OpenTelemetry.
With traces, you can see one continuous timeline of traffic events across different pods and services. This can help you identify bottlenecks and conduct root-cause analysis.
Spans show one part of the journey. The attributes attached to those spans give you the contextual metadata that you need to have searchable and meaningful traces. The attributes are simple key: value pairs attached to each span.
The MCP gateway uses the following router spans:
| Span | When | Description |
|---|---|---|
|
|
Every | Root span. Starts when request headers arrive, ends after response headers are processed. |
|
| The request body is parsed |
Routing decision: |
|
|
Non- |
Pass-through to broker: |
|
|
| Full tool call handling including session and server resolution. |
|
|
Inside | Queries the broker to resolve which backend MCP server owns the tool. |
|
|
Inside | Queries the session cache to look up an existing backend session. |
|
|
Cache miss during |
Hairpin |
|
|
After | Persists the new backend session in the session cache. |
The attributes in the following tables use OpenTelemetry MCP Semantic Conventions.
| Attribute | Source | Description |
|---|---|---|
|
|
|
HTTP method, |
|
|
|
Request path, |
|
|
| Envoy request ID |
|
|
JSON-RPC |
MCP method, |
|
|
JSON-RPC |
Tool name, only for |
|
|
JSON-RPC | JSON-RPC request ID |
|
|
JSON-RPC | Always "2.0" |
|
|
JSON-RPC |
Same as |
|
|
| Gateway session ID |
|
|
| Client IP address |
|
|
| Response status code |
| Attribute | Description |
|---|---|
|
| MCP method |
|
|
Routing decision: |
| Attribute | Description |
|---|---|
|
| Tool name from the request |
|
| Gateway session ID |
|
| Resolved backend server name |
|
| Resolved backend server hostname |
| Attribute | Description |
|---|---|
|
|
Error classification, such as |
|
| Component that generated the error, such as`ext-proc` |
|
| HTTP status code returned |
When there is an error, spans include these attributes.
2.2. The MCP gateway audit trail Copy linkLink copied to clipboard!
You can use an audit trail for compliance and accountability that captures caller identity, tool names, and Model Context Protocol (MCP) session context through the MCP gateway.
Users connect to backend MCP servers and run functions, for example, reading a database, running a script, or editing files. By configuring an audit trail that comes from your AuthPolicy objects and the Istio Telemetry API, you can configure JSON access logging on your Gateway object and discover who did what and when.
MCP gateway sets the following routing headers on every request:
-
x-mcp-method -
x-mcp-toolname -
x-mcp-servername -
mcp-session-id
These headers are available to any Envoy access log through %REQ(…)% format strings. With configuration, you can add the following username header and structured access logs:
-
x-auth-identity: Adds caller identity. After you configure the header, theAuthPolicyobject validates JWT tokens and injects the authenticated username as the request header. -
With an Istio Telemetry resource, you can configure JSON access logging on the
Gatewayobject, capturing MCP routing headers and the identity header.
The result is a JSON access log on the MCP Gateway object pod’s standard output (stdout) that tells you who called what tool, on which server, during which session, and at what time.
2.2.1. Configure an AuthPolicy with identity injection Copy linkLink copied to clipboard!
To capture authenticated user identities in access logs for all request types, you must create an AuthPolicy custom resource (CR) for each of the MCP gateway two listeners.
The MCP gateway uses the two following listeners:
-
mcp: This listener handles client requests such asinitializeandtools/list. -
mcps: This listener handles tool call routing to the backend Model Context Protocol (MCP) servers.
When you have configured the AuthPolicy CRs, after Authorino validates the JWT, it extracts the preferred_username field value from the token claims and injects it as the x-auth-identity request header. This header is trustworthy because Authorino strips any client-supplied value and sets it from the validated token. With both policies in place, the mcp_user_id field is displayed in access log entries for all MCP request types.
Prerequisites
- You installed the MCP gateway.
- You installed Connectivity Link.
-
You created a
Gatewayobject. - You set up an identity provider.
- You configured Istio as the Gateway API provider.
Procedure
Create an
AuthPolicyCR for the client-facing listener,mcp, by using the following example:Client-facing listener AuthPolicy CR
apiVersion: kuadrant.io/v1 kind: AuthPolicy metadata: name: <audit_auth_client_listener> namespace: <gateway_namespace> spec: targetRef: group: gateway.networking.k8s.io kind: Gateway name: <mcp_gateway> sectionName: mcp defaults: when: - predicate: "!request.path.contains('/.well-known')" rules: authentication: 'keycloak': jwt: issuerUrl: https://keycloak.example.com:8002/realms/mcp response: success: headers: "x-auth-identity": plain: selector: auth.identity.preferred_username unauthenticated: code: 401 headers: 'WWW-Authenticate': value: Bearer resource_metadata=http://mcp.example.com:8001/.well-known/oauth-protected-resource/mcp body: value: | { "error": "Unauthorized", "message": "Authentication required." }-
Replace
<mcp_gateway>with the name of your MCP gateway deployment. Replace
<gateway_namespace>with the namespace where you applied yourGatewayobject.NoteThis
mcplistenerAuthPolicyCR validates JSON web tokens (JWTs) on client requests and injects the authenticated username as a request header.
-
Replace
Apply the client-listener
AuthPolicyCR by running the following command:$ oc apply -f <audit_auth_client_listener.yaml>Replace
<audit_auth_client_listener.yaml>with the filename that you used.Create an
AuthPolicyCR for the backend-facing listener,mcps, by using the following example:Client-facing listener
AuthPolicyCRapiVersion: kuadrant.io/v1 kind: AuthPolicy metadata: name: <audit_auth_backend_listener> namespace: <gateway_namespace> spec: targetRef: group: gateway.networking.k8s.io kind: Gateway name: <mcp_gateway> sectionName: mcps defaults: rules: authentication: 'keycloak': jwt: issuerUrl: https://keycloak.example.com:8002/realms/mcp response: success: headers: "x-auth-identity": plain: selector: auth.identity.preferred_username unauthenticated: code: 401 headers: 'WWW-Authenticate': value: Bearer resource_metadata=http://mcp.example.com:8001/.well-known/oauth-protected-resource/mcp body: value: | { "error": "Unauthorized", "message": "Authentication required." }-
Replace
<mcp_gateway>with the name of your MCP gateway deployment. -
Replace
<gateway_namespace>with the namespace where you applied yourGatewayobject. Replace the
preferred_usernamefield value withsubor another claim depending on both your identity provider and what you want as the audit identity. For Red Hat build of Keycloak, thepreferred_usernamefield value gives a human-readable username but requires theprofilescope in the token request.NoteThis
mcpslistener policy validates the same JWT on tool call requests routed to backend servers and injects the identity header there, too.
-
Replace
Apply the backend-listener
AuthPolicyCR by running the following command:$ oc apply -f <audit_auth_backend_listener.yaml>Replace
<audit_auth_backend_listener.yaml>with the filename that you used.
2.2.2. Verify AuthPolicy identity injection Copy linkLink copied to clipboard!
Verify that the identity header you created in your audit AuthPolicy custom resource (CR) is being injected by calling a tool and checking what the backend receives.
Prerequisites
- You installed the MCP gateway.
- You installed Connectivity Link.
-
You created a
Gatewayobject. - You set up an identity provider.
- You configured Istio as the Gateway API provider.
-
You created
AuthPolicyaudit CRs for both MCP gateway listeners.
Procedure
Get a token by running the following command:
$ TOKEN=$(curl -s -X POST "https://keycloak.example.com:8002/realms/mcp/protocol/openid-connect/token" \ -d "grant_type=password&client_id=mcp-gateway&client_secret=secret&username=mcp&password=mcp&scope=openid+profile" \ | jq -r '.access_token')Adjust this command for your identity provider.
Initialize a session by running the following command:
$ SESSION_ID=$(curl -si http://mcp.example.com:8001/mcp \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"audit-test","version":"0.0.1"}}}' \ | grep -i 'mcp-session-id:' | awk '{print $2}' | tr -d '\r')The expected output is the session ID returned in the response header.
Call the headers tool to see which headers reach the backend by running the following command:
$ curl -s http://mcp.example.com:8001/mcp \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "Mcp-Session-Id: $SESSION_ID" \ -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"test1_headers","arguments":{}}}' \ | jq '.result.content[0].text' | grep -i "x-auth-identity"The expected output is the
x-auth-identityheader that lists the Keycloak username in the response.
2.2.3. Add a custom audit log provider Copy linkLink copied to clipboard!
Define the JSON format for the audit logs by adding a custom access log provider to Istio’s MeshConfig structure that dictates all the global settings for the mesh. Istio acts as the controller that translates your resources into Envoy configurations.
The approach used in the following procedures only works if you are using OpenShift Service Mesh. If you are using the OpenShift Container Platform default Ingress Operator to manage Istio objects, changes are reconciled back to the previous settings.
Prerequisites
- You installed the MCP gateway.
- You installed Connectivity Link.
-
You created a
Gatewayobject. - You set up an identity provider.
- You configured Istio as the Gateway API provider.
-
You created
AuthPolicyaudit CRs for both MCP gateway listeners.
Procedure
Edit the Istio object by using the following example:
Example Istio custom access log provider
apiVersion: operator.istio.io/v1alpha1 kind: Istio metadata: name: default namespace: istio-system spec: meshConfig: extensionProviders: - envoyFileAccessLog: logFormat: labels: bytes_received: '%BYTES_RECEIVED%' bytes_sent: '%BYTES_SENT%' duration_ms: '%DURATION%' mcp_method: '%REQ(X-MCP-METHOD)%' mcp_server_name: '%REQ(X-MCP-SERVERNAME)%' mcp_session_id: '%REQ(MCP-SESSION-ID)%' mcp_tool_name: '%REQ(X-MCP-TOOLNAME)%' mcp_user_id: '%REQ(X-AUTH-IDENTITY)%' method: '%REQ(:METHOD)%' path: '%REQ(:PATH)%' request_id: '%REQ(X-REQUEST-ID)%' response_code: '%RESPONSE_CODE%' timestamp: '%START_TIME%' traceparent: '%REQ(TRACEPARENT)%' upstream_host: '%UPSTREAM_HOST%' path: /dev/stdout name: mcp-json-access-log #...ImportantIf you have other extension providers configured, for example, OpenTelemetry tracing, include them in the
extensionProvidersarray with the access log provider. The merge of the resources replaces the array.
2.2.3.1. Custom audit log provider fields Copy linkLink copied to clipboard!
Understand the MCP gateway audit log fields by using the reference table for descriptions of each field and the source and type of information that each provides.
| Field | Source | Description |
|---|---|---|
|
|
| The start time of the session |
|
|
|
The HTTP method of the request, for example, |
|
|
| The HTTP request path, including any query parameters |
|
|
| The HTTP response status code returned to the client |
|
|
| Unique string identifier for tracking the end-to-end request lifecycle |
|
|
| W3C Trace Context for cross-system correlation |
|
|
|
MCP method, such as |
|
|
| Tool name after prefix stripping |
|
|
| Backend MCP server name |
|
|
| MCP session identifier |
|
|
|
Authenticated user identity, injected by an |
|
|
| Total duration of the request in milliseconds, from the start time to the last byte sent |
|
|
| The upstream host IP address and port that handled the request |
|
|
| Body bytes sent downstream to the client |
|
|
| Body bytes received upstream from the client |
2.2.4. Enable the access log on the Gateway pods workload Copy linkLink copied to clipboard!
After you update your Istio custom resource (CR), you must create a Telemetry CR in the namespace where your Gateway object is applied to enable the access log on the Gateway pods workload.
The resource created in this example scopes the access log to the <mcp_gateway> Gateway pods only. Other workloads in the mesh are not affected.
Prerequisites
- You installed the MCP gateway.
- You installed Connectivity Link.
-
You created a
Gatewayobject. - You set up an identity provider.
- You configured Istio as the Gateway API provider.
-
You created
AuthPolicyaudit CRs for both MCP gateway listeners. - You added your custom log provider.
Procedure
Create a
TelemetryCR in the namespace where yourGatewayobject is applied by using the following example:Example Telemetry CR for audit logs
apiVersion: telemetry.istio.io/v1 kind: Telemetry metadata: name: <mcp_audit_logging> namespace: <gateway_namespace> spec: selector: matchLabels: gateway.networking.k8s.io/gateway-name: <mcp_gateway> accessLogging: - providers: - name: mcp-json-access-log-
Replace
<mcp_audit_logging>with the name of your audit loggingTelemetryCR. -
Replace
<gateway_namespace>with the namespace where you applied yourGatewayobject. -
Replace
<mcp_gateway>with the name of your MCP gateway deployment.
-
Replace
Apply the
TelemetryCR by running the following command:$ oc apply -f <mcp_audit_logging.yaml>Replace
<mcp_audit_logging.yaml>with the name of your audit logging YAML.
Verification
Verify that your
TelemetryCR is applied by running the following command:$ oc get telemetry -n <gateway_namespace>Replace
<gateway_namespace>with the namespace where you applied yourGatewayobject.
2.2.5. Verify your audit trail Copy linkLink copied to clipboard!
After you have updated your Istio custom resource (CR), you must create a Telemetry CR in the namespace where your Gateway object is applied to enable the access log on the gateway’s workload.
The resource created in this example scopes the access log to the <mcp_gateway> Gateway object pods only. Other workloads in the mesh are not affected.
Prerequisites
- You installed the MCP gateway.
- You installed Connectivity Link.
-
You created a
Gatewayobject. - You set up an identity provider.
- You configured Istio as the Gateway API provider.
-
You created
AuthPolicyaudit custom resources (CRs) for both MCP gateway listeners. - You added your custom log provider.
-
You created your
TelemetryCR.
Procedure
Make an authenticated tool call by reusing both the token and session ID from the earlier step by running the following command:
$ curl -s http://mcp.example.com:8001/mcp \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "Mcp-Session-Id: $SESSION_ID" \ -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"test1_greet","arguments":{"name":"audit-test"}}}'Check the
Gatewayobject pod logs for the JSON access log entry by running the following command:$ oc logs -n <gateway_namespace> -l gateway.networking.k8s.io/gateway-name=<mcp_gateway> --since=30s \ | grep '"mcp_method"' | tail -1 | jq .-
Replace
<gateway_namespace>with the namespace where you applied yourGatewayobject. Replace
<mcp_gateway>with the name of your MCP gateway deployment.Example output
{ "timestamp": "2026-05-21T14:23:01.123Z", "method": "POST", "path": "/mcp", "response_code": 200, "request_id": "abc-111", "traceparent": null, "mcp_method": "tools/call", "mcp_tool_name": "greet", "mcp_server_name": "mcp-test/test-server1", "mcp_session_id": "sess-7a3b", "mcp_user_id": "mcp", "duration_ms": 342, "upstream_host": "10.0.1.5:8080", "bytes_sent": 1024, "bytes_received": 512 }
-
Replace
2.2.6. Use an audit trail without authentication Copy linkLink copied to clipboard!
If you do not have an authentication layer, the routing headers still offer useful audit context.
The mcp_user_id field is empty (-), but you can still see the following details:
- Which tool a user called
- On which server the call took place
- In which session the call took place
- At what time the call took place
- How long the interaction lasted
You can configure the x-auth-identity header name and the JSON web token (JWT) claim in the AuthPolicy CR. Adjust the spec.defaults.rules.response.success.headers section to match your identity provider according to the following list:
-
Red Hat build of Keycloak:
auth.identity.preferred_usernameorauth.identity.email. -
Generic OIDC:
auth.identity.sub; this is the subject claim, always present in JWTs. -
Custom claims:
auth.identity.<claim_name>for any claim in the JWT payload.
If you change the header name, you must also update the mcp_user_id field in your MeshConfig extension provider to match the %REQ(<HEADER_NAME>)% format.
Generic OIDC AuthPolicy CR header configuration example
apiVersion: kuadrant.io/v1
kind: AuthPolicy
metadata:
name: <audit_no_auth_client_listener>
namespace: <gateway_namespace>
spec:
#...
defaults:
#...
rules:
#...
response:
success:
headers:
"x-auth-identity":
plain:
selector: auth.identity.sub
#...