Chapter 2. Observe MCP gateway connections


You can use observability in MCP gateway to debug silent failures, track destructive annotations and latencies, and configure an audit trail.

The MCP gateway uses Red Hat build of OpenTelemetry throughout all components to give you consistent logging, distributed tracing, and metrics. By using observability, you can achieve the following goals:

  • Discover which component generated an error, such as Envoy, an MCP gateway component, or a backend MCP server.
  • Understand how requests flow through the system components.
  • See which tools were called and which MCP servers executed those tools.
  • Understand where to find relevant logs for each component.
  • Examine metrics about tool call patterns, success rates, and error rates.
  • Examine distributed tracing across the request path.

Prerequisites

  • You installed MCP gateway.
  • You installed Connectivity Link.
  • You configured a Gateway object.
  • You are logged into a running OpenShift Container Platform cluster with an admin role.
  • You installed Red Hat build of OpenTelemetry.
  • You configured the Red Hat OpenShift Distributed Tracing Platform for traces.
  • You installed the OpenShift Logging Operator (Loki).
  • You configured an S3-compatible persistent storage volume for Loki to store long-term MCP tool call logs.

Procedure

  1. Use the following example YAML to create a collector in your MCP gateway deployment namespace:

    Example OpenTelemetryCollector custom resource (CR)

    apiVersion: opentelemetry.io/v1beta1
    kind: OpenTelemetryCollector
    metadata:
      name: <mcp_otel_collector>
      namespace: <mcp_gateway_system>
    spec:
      mode: deployment
      config:
        receivers:
          otlp:
            protocols:
              http:
                endpoint: 0.0.0.0:4318
              grpc:
                endpoint: 0.0.0.0:4317
        exporters:
          otlp/tempo:
            endpoint: tempo-gateway-http.openshift-tracing.svc.cluster.local:4317
            tls:
              insecure: true
          debug:
            verbosity: basic
        service:
          pipelines:
            traces:
              receivers: [otlp]
              exporters: [otlp/tempo, debug]
            logs:
              receivers: [otlp]
              exporters: [debug]

    • Replace the value of metadata.name: with the name you want to assign to this collector.
    • Replace the value of metadata.namespace: with the namespace of your MCP gateway deployment.
    • The spec.config.exporters: field value points to distributed tracing.
    • The setting, spec.config.exporters.otlp/tempo.tls.insecure: true is for internal cluster communication without TLS.
  2. Apply the following environment variables to your OpenShift Container Platform cluster by running the following command:

    $ oc set env deployment/mcp-gateway \
      OTEL_EXPORTER_OTLP_ENDPOINT="http://<mcp_otel_collector>-collector.<mcp_gateway_system>.svc.cluster.local:4318" \
      OTEL_EXPORTER_OTLP_INSECURE="true" \
      OTEL_SERVICE_NAME="mcp-gateway"
    • Replace <mcp_otel_collector> with the name you used in your OpenTelemetryCollector CR. When Red Hat build of OpenTelemetry creates a collector, the service name is [metadata.name]-collector.
    • Replace <mcp_gateway_system> with the namespace of your MCP gateway deployment.
  3. Optional. If you want to send traces to one collector and logs to a different one, set the following additional environment variables:

    $ oc set env deployment/mcp-gateway \
      OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://trace-collector.tracing-namespace.svc:4317" \
      OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="http://log-collector.logging-namespace.svc:4317"
    • Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to override the endpoint for traces.
    • Set OTEL_EXPORTER_OTLP_LOGS_ENDPOINT to override the endpoint for logs.
  4. If you are using Red Hat OpenShift Distributed Tracing Platform, select your stack name as the data source in the OpenShift Web Console Observe > Traces dashboard. For example, tempo-mcp.
  5. After you select the stack name, you can consult an attribute in your OpenShift Web Console OpenShift Observe > Traces dashboard to find where errors have occurred or to gather information for trend identification.

    • You can also filter by service.name="mcp-gateway".
  6. When using Loki, use the Observe > Logs view and toggle the Structured view to filter by mcp.session.id.
  7. When using Loki, use the Trace timeline view, look for the Log icon next to spans to jump directly to the relevant logs.

Troubleshooting

  • If you do not see any traces, check if the NetworkPolicy CR in your namespace allows traffic from the MCP gateway to the Red Hat build of OpenTelemetry collector service.

2.1.1. MCP gateway router spans for observability

When enabled, the Model Context Protocol (MCP) router, ext_proc, emits trace spans for every request and can export structured logs through Red Hat build of OpenTelemetry.

With traces, you can see one continuous timeline of traffic events across different pods and services. This can help you identify bottlenecks and conduct root-cause analysis.

Spans show one part of the journey. The attributes attached to those spans give you the contextual metadata that you need to have searchable and meaningful traces. The attributes are simple key: value pairs attached to each span.

The MCP gateway uses the following router spans:

Table 2.1. MCP router spans
SpanWhenDescription

mcp-router.process

Every ext_proc stream

Root span. Starts when request headers arrive, ends after response headers are processed.

mcp-router.route-decision

The request body is parsed

Routing decision: tool-call versus pass-through to broker.

mcp-router.broker-passthrough

Non-tool-call requests

Pass-through to broker: initialize, tools/list, notifications. No child spans means the broker is not instrumented.

mcp-router.tool-call

tools/call requests

Full tool call handling including session and server resolution.

mcp-router.broker.get-server-info

Inside tool-call

Queries the broker to resolve which backend MCP server owns the tool.

mcp-router.session-cache.get

Inside tool-call

Queries the session cache to look up an existing backend session.

mcp-router.session-init

Cache miss during tool-call

Hairpin initialize request through the gateway to the backend MCP server.

mcp-router.session-cache.store

After session-init

Persists the new backend session in the session cache.

The attributes in the following tables use OpenTelemetry MCP Semantic Conventions.

Table 2.2. MCP router root span, mcp-router.process, attributes
AttributeSourceDescription

http.method

:method header

HTTP method, POST

http.path

:path header

Request path, /mcp

http.request_id

x-request-id header

Envoy request ID

mcp.method.name

JSON-RPC method field

MCP method, initialize, tools/call, tools/list, and so on

gen_ai.tool.name

JSON-RPC params.name

Tool name, only for tools/call

jsonrpc.request.id

JSON-RPC id field

JSON-RPC request ID

jsonrpc.protocol.version

JSON-RPC jsonrpc field

Always "2.0"

gen_ai.operation.name

JSON-RPC method field

Same as mcp.method.name

mcp.session.id

mcp-session-id header

Gateway session ID

client.address

x-forwarded-for header

Client IP address

http.status_code

:status response header

Response status code

Table 2.3. MCP route decision span, mcp-router.route-decision, attributes
AttributeDescription

mcp.method.name

MCP method

mcp.route

Routing decision: tool-call, broker, or elicitation-response

Table 2.4. MCP tool call span, mcp-router.tool-call, attributes
AttributeDescription

gen_ai.tool.name

Tool name from the request

mcp.session.id

Gateway session ID

mcp.server

Resolved backend server name

mcp.server.hostname

Resolved backend server hostname

Table 2.5. MCP error attributes
AttributeDescription

error.type

Error classification, such as tool_not_found, missing_tool_name, invalid_session, session_cache_error, session_init_error, marshal_error, path_parse_error

error_source

Component that generated the error, such as`ext-proc`

http.status_code

HTTP status code returned

When there is an error, spans include these attributes.

2.2. The MCP gateway audit trail

You can use an audit trail for compliance and accountability that captures caller identity, tool names, and Model Context Protocol (MCP) session context through the MCP gateway.

Users connect to backend MCP servers and run functions, for example, reading a database, running a script, or editing files. By configuring an audit trail that comes from your AuthPolicy objects and the Istio Telemetry API, you can configure JSON access logging on your Gateway object and discover who did what and when.

MCP gateway sets the following routing headers on every request:

  • x-mcp-method
  • x-mcp-toolname
  • x-mcp-servername
  • mcp-session-id

These headers are available to any Envoy access log through %REQ(…​)% format strings. With configuration, you can add the following username header and structured access logs:

  • x-auth-identity: Adds caller identity. After you configure the header, the AuthPolicy object validates JWT tokens and injects the authenticated username as the request header.
  • With an Istio Telemetry resource, you can configure JSON access logging on the Gateway object, capturing MCP routing headers and the identity header.

The result is a JSON access log on the MCP Gateway object pod’s standard output (stdout) that tells you who called what tool, on which server, during which session, and at what time.

To capture authenticated user identities in access logs for all request types, you must create an AuthPolicy custom resource (CR) for each of the MCP gateway two listeners.

The MCP gateway uses the two following listeners:

  • mcp: This listener handles client requests such as initialize and tools/list.
  • mcps: This listener handles tool call routing to the backend Model Context Protocol (MCP) servers.

When you have configured the AuthPolicy CRs, after Authorino validates the JWT, it extracts the preferred_username field value from the token claims and injects it as the x-auth-identity request header. This header is trustworthy because Authorino strips any client-supplied value and sets it from the validated token. With both policies in place, the mcp_user_id field is displayed in access log entries for all MCP request types.

Prerequisites

  • You installed the MCP gateway.
  • You installed Connectivity Link.
  • You created a Gateway object.
  • You set up an identity provider.
  • You configured Istio as the Gateway API provider.

Procedure

  1. Create an AuthPolicy CR for the client-facing listener, mcp, by using the following example:

    Client-facing listener AuthPolicy CR

    apiVersion: kuadrant.io/v1
    kind: AuthPolicy
    metadata:
      name: <audit_auth_client_listener>
      namespace: <gateway_namespace>
    spec:
      targetRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: <mcp_gateway>
        sectionName: mcp
      defaults:
        when:
          - predicate: "!request.path.contains('/.well-known')"
        rules:
          authentication:
            'keycloak':
              jwt:
                issuerUrl: https://keycloak.example.com:8002/realms/mcp
          response:
            success:
              headers:
                "x-auth-identity":
                  plain:
                    selector: auth.identity.preferred_username
            unauthenticated:
              code: 401
              headers:
                'WWW-Authenticate':
                  value: Bearer resource_metadata=http://mcp.example.com:8001/.well-known/oauth-protected-resource/mcp
              body:
                value: |
                  {
                    "error": "Unauthorized",
                    "message": "Authentication required."
                  }

    • Replace <mcp_gateway> with the name of your MCP gateway deployment.
    • Replace <gateway_namespace> with the namespace where you applied your Gateway object.

      Note

      This mcp listener AuthPolicy CR validates JSON web tokens (JWTs) on client requests and injects the authenticated username as a request header.

  2. Apply the client-listener AuthPolicy CR by running the following command:

    $ oc apply -f <audit_auth_client_listener.yaml>

    Replace <audit_auth_client_listener.yaml> with the filename that you used.

  3. Create an AuthPolicy CR for the backend-facing listener, mcps, by using the following example:

    Client-facing listener AuthPolicy CR

    apiVersion: kuadrant.io/v1
    kind: AuthPolicy
    metadata:
      name: <audit_auth_backend_listener>
      namespace: <gateway_namespace>
    spec:
      targetRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: <mcp_gateway>
        sectionName: mcps
      defaults:
        rules:
          authentication:
            'keycloak':
              jwt:
                issuerUrl: https://keycloak.example.com:8002/realms/mcp
          response:
            success:
              headers:
                "x-auth-identity":
                  plain:
                    selector: auth.identity.preferred_username
            unauthenticated:
              code: 401
              headers:
                'WWW-Authenticate':
                  value: Bearer resource_metadata=http://mcp.example.com:8001/.well-known/oauth-protected-resource/mcp
              body:
                value: |
                  {
                    "error": "Unauthorized",
                    "message": "Authentication required."
                  }

    • Replace <mcp_gateway> with the name of your MCP gateway deployment.
    • Replace <gateway_namespace> with the namespace where you applied your Gateway object.
    • Replace the preferred_username field value with sub or another claim depending on both your identity provider and what you want as the audit identity. For Red Hat build of Keycloak, the preferred_username field value gives a human-readable username but requires the profile scope in the token request.

      Note

      This mcps listener policy validates the same JWT on tool call requests routed to backend servers and injects the identity header there, too.

  4. Apply the backend-listener AuthPolicy CR by running the following command:

    $ oc apply -f <audit_auth_backend_listener.yaml>

    Replace <audit_auth_backend_listener.yaml> with the filename that you used.

2.2.2. Verify AuthPolicy identity injection

Verify that the identity header you created in your audit AuthPolicy custom resource (CR) is being injected by calling a tool and checking what the backend receives.

Prerequisites

  • You installed the MCP gateway.
  • You installed Connectivity Link.
  • You created a Gateway object.
  • You set up an identity provider.
  • You configured Istio as the Gateway API provider.
  • You created AuthPolicy audit CRs for both MCP gateway listeners.

Procedure

  1. Get a token by running the following command:

    $ TOKEN=$(curl -s -X POST "https://keycloak.example.com:8002/realms/mcp/protocol/openid-connect/token" \
      -d "grant_type=password&client_id=mcp-gateway&client_secret=secret&username=mcp&password=mcp&scope=openid+profile" \
      | jq -r '.access_token')

    Adjust this command for your identity provider.

  2. Initialize a session by running the following command:

    $ SESSION_ID=$(curl -si http://mcp.example.com:8001/mcp \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"audit-test","version":"0.0.1"}}}' \
      | grep -i 'mcp-session-id:' | awk '{print $2}' | tr -d '\r')

    The expected output is the session ID returned in the response header.

  3. Call the headers tool to see which headers reach the backend by running the following command:

    $ curl -s http://mcp.example.com:8001/mcp \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -H "Mcp-Session-Id: $SESSION_ID" \
      -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"test1_headers","arguments":{}}}' \
      | jq '.result.content[0].text' | grep -i "x-auth-identity"

    The expected output is the x-auth-identity header that lists the Keycloak username in the response.

2.2.3. Add a custom audit log provider

Define the JSON format for the audit logs by adding a custom access log provider to Istio’s MeshConfig structure that dictates all the global settings for the mesh. Istio acts as the controller that translates your resources into Envoy configurations.

Important

The approach used in the following procedures only works if you are using OpenShift Service Mesh. If you are using the OpenShift Container Platform default Ingress Operator to manage Istio objects, changes are reconciled back to the previous settings.

Prerequisites

  • You installed the MCP gateway.
  • You installed Connectivity Link.
  • You created a Gateway object.
  • You set up an identity provider.
  • You configured Istio as the Gateway API provider.
  • You created AuthPolicy audit CRs for both MCP gateway listeners.

Procedure

  • Edit the Istio object by using the following example:

    Example Istio custom access log provider

    apiVersion: operator.istio.io/v1alpha1
    kind: Istio
    metadata:
      name: default
      namespace: istio-system
    spec:
      meshConfig:
        extensionProviders:
          - envoyFileAccessLog:
              logFormat:
                labels:
                  bytes_received: '%BYTES_RECEIVED%'
                  bytes_sent: '%BYTES_SENT%'
                  duration_ms: '%DURATION%'
                  mcp_method: '%REQ(X-MCP-METHOD)%'
                  mcp_server_name: '%REQ(X-MCP-SERVERNAME)%'
                  mcp_session_id: '%REQ(MCP-SESSION-ID)%'
                  mcp_tool_name: '%REQ(X-MCP-TOOLNAME)%'
                  mcp_user_id: '%REQ(X-AUTH-IDENTITY)%'
                  method: '%REQ(:METHOD)%'
                  path: '%REQ(:PATH)%'
                  request_id: '%REQ(X-REQUEST-ID)%'
                  response_code: '%RESPONSE_CODE%'
                  timestamp: '%START_TIME%'
                  traceparent: '%REQ(TRACEPARENT)%'
                  upstream_host: '%UPSTREAM_HOST%'
              path: /dev/stdout
            name: mcp-json-access-log
    #...

    Important

    If you have other extension providers configured, for example, OpenTelemetry tracing, include them in the extensionProviders array with the access log provider. The merge of the resources replaces the array.

2.2.3.1. Custom audit log provider fields

Understand the MCP gateway audit log fields by using the reference table for descriptions of each field and the source and type of information that each provides.

Table 2.6. Audit log fields
FieldSourceDescription

timestamp

%START_TIME%

The start time of the session

method

%REQ(:METHOD)%

The HTTP method of the request, for example, GET, POST

path

%REQ(:PATH)%

The HTTP request path, including any query parameters

response_code

%RESPONSE_CODE%

The HTTP response status code returned to the client

request_id

%REQ(X-REQUEST-ID)%

Unique string identifier for tracking the end-to-end request lifecycle

traceparent

%REQ(TRACEPARENT)%

W3C Trace Context for cross-system correlation

mcp_method

%REQ(X-MCP-METHOD)%

MCP method, such as tools/call, tools/list, initialize, and so on

mcp_tool_name

%REQ(X-MCP-TOOLNAME)%

Tool name after prefix stripping

mcp_server_name

%REQ(X-MCP-SERVERNAME)%

Backend MCP server name

mcp_session_id

%REQ(MCP-SESSION-ID)%

MCP session identifier

mcp_user_id

%REQ(X-AUTH-IDENTITY)%

Authenticated user identity, injected by an AuthPolicy custom resource

duration_ms

%DURATION%

Total duration of the request in milliseconds, from the start time to the last byte sent

upstream_host

%UPSTREAM_HOST%

The upstream host IP address and port that handled the request

bytes_sent

%BYTES_SENT%

Body bytes sent downstream to the client

bytes_received

%BYTES_RECEIVED%

Body bytes received upstream from the client

After you update your Istio custom resource (CR), you must create a Telemetry CR in the namespace where your Gateway object is applied to enable the access log on the Gateway pods workload.

The resource created in this example scopes the access log to the <mcp_gateway> Gateway pods only. Other workloads in the mesh are not affected.

Prerequisites

  • You installed the MCP gateway.
  • You installed Connectivity Link.
  • You created a Gateway object.
  • You set up an identity provider.
  • You configured Istio as the Gateway API provider.
  • You created AuthPolicy audit CRs for both MCP gateway listeners.
  • You added your custom log provider.

Procedure

  1. Create a Telemetry CR in the namespace where your Gateway object is applied by using the following example:

    Example Telemetry CR for audit logs

    apiVersion: telemetry.istio.io/v1
    kind: Telemetry
    metadata:
      name: <mcp_audit_logging>
      namespace: <gateway_namespace>
    spec:
      selector:
        matchLabels:
          gateway.networking.k8s.io/gateway-name: <mcp_gateway>
      accessLogging:
        - providers:
            - name: mcp-json-access-log

    • Replace <mcp_audit_logging> with the name of your audit logging Telemetry CR.
    • Replace <gateway_namespace> with the namespace where you applied your Gateway object.
    • Replace <mcp_gateway> with the name of your MCP gateway deployment.
  2. Apply the Telemetry CR by running the following command:

    $ oc apply -f <mcp_audit_logging.yaml>

    Replace <mcp_audit_logging.yaml> with the name of your audit logging YAML.

Verification

  1. Verify that your Telemetry CR is applied by running the following command:

    $ oc get telemetry -n <gateway_namespace>

    Replace <gateway_namespace> with the namespace where you applied your Gateway object.

2.2.5. Verify your audit trail

After you have updated your Istio custom resource (CR), you must create a Telemetry CR in the namespace where your Gateway object is applied to enable the access log on the gateway’s workload.

The resource created in this example scopes the access log to the <mcp_gateway> Gateway object pods only. Other workloads in the mesh are not affected.

Prerequisites

  • You installed the MCP gateway.
  • You installed Connectivity Link.
  • You created a Gateway object.
  • You set up an identity provider.
  • You configured Istio as the Gateway API provider.
  • You created AuthPolicy audit custom resources (CRs) for both MCP gateway listeners.
  • You added your custom log provider.
  • You created your Telemetry CR.

Procedure

  1. Make an authenticated tool call by reusing both the token and session ID from the earlier step by running the following command:

    $ curl -s http://mcp.example.com:8001/mcp \
      -H "Authorization: Bearer $TOKEN" \
      -H "Content-Type: application/json" \
      -H "Mcp-Session-Id: $SESSION_ID" \
      -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"test1_greet","arguments":{"name":"audit-test"}}}'
  2. Check the Gateway object pod logs for the JSON access log entry by running the following command:

    $ oc logs -n <gateway_namespace> -l gateway.networking.k8s.io/gateway-name=<mcp_gateway> --since=30s \
      | grep '"mcp_method"' | tail -1 | jq .
    • Replace <gateway_namespace> with the namespace where you applied your Gateway object.
    • Replace <mcp_gateway> with the name of your MCP gateway deployment.

      Example output

      {
        "timestamp": "2026-05-21T14:23:01.123Z",
        "method": "POST",
        "path": "/mcp",
        "response_code": 200,
        "request_id": "abc-111",
        "traceparent": null,
        "mcp_method": "tools/call",
        "mcp_tool_name": "greet",
        "mcp_server_name": "mcp-test/test-server1",
        "mcp_session_id": "sess-7a3b",
        "mcp_user_id": "mcp",
        "duration_ms": 342,
        "upstream_host": "10.0.1.5:8080",
        "bytes_sent": 1024,
        "bytes_received": 512
      }

2.2.6. Use an audit trail without authentication

If you do not have an authentication layer, the routing headers still offer useful audit context.

The mcp_user_id field is empty (-), but you can still see the following details:

  • Which tool a user called
  • On which server the call took place
  • In which session the call took place
  • At what time the call took place
  • How long the interaction lasted

You can configure the x-auth-identity header name and the JSON web token (JWT) claim in the AuthPolicy CR. Adjust the spec.defaults.rules.response.success.headers section to match your identity provider according to the following list:

  • Red Hat build of Keycloak: auth.identity.preferred_username or auth.identity.email.
  • Generic OIDC: auth.identity.sub; this is the subject claim, always present in JWTs.
  • Custom claims: auth.identity.<claim_name> for any claim in the JWT payload.
Important

If you change the header name, you must also update the mcp_user_id field in your MeshConfig extension provider to match the %REQ(<HEADER_NAME>)% format.

Generic OIDC AuthPolicy CR header configuration example

apiVersion: kuadrant.io/v1
kind: AuthPolicy
metadata:
  name: <audit_no_auth_client_listener>
  namespace: <gateway_namespace>
spec:
#...
  defaults:
#...
    rules:
#...
      response:
        success:
          headers:
            "x-auth-identity":
              plain:
                selector: auth.identity.sub
#...

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

Legal Notice

Theme

© 2026 Red Hat
Back to top