Configure Telemetry Gateway forwarding (optional)

By default, the Telemetry Gateway exposes device metrics for local Prometheus scraping. You can forward that telemetry to a centralized OTLP gRPC backend over mutual TLS to aggregate data across multiple Red Hat Edge Manager deployments.

Before you begin

  • Red Hat Edge Manager is deployed on the cluster.
  • Cluster administrator permissions and the OpenShift CLI (oc) installed and authenticated.
  • A client certificate, private key, and upstream CA certificate file are available for mutual TLS with the upstream OTLP gRPC endpoint.
  • The upstream OTLP gRPC endpoint address and port are known.

About this task

Consider forwarding when you:

  • Already use a centralized observability platform (for example, Grafana Cloud, Datadog, or New Relic).
  • Want to aggregate telemetry from multiple Red Hat Edge Manager deployments.
  • Must integrate with organization-wide monitoring systems.

To configure forwarding with mutual TLS (mTLS), use the following procedure.

The Flight Control Helm chart does not expose Telemetry Gateway forwarding through Helm values. The chart values schema only accepts telemetryGateway.image and telemetryGateway.additionalRouteLabels, and the chart template hardcodes listen and export settings in the flightctl-telemetry-gateway-configConfigMap. Configure forwarding by editing that ConfigMap and patching the Telemetry Gateway Deployment.

Note

The following steps use manually created certificates for testing. For production clusters, consider using cert-manager to issue and rotate certificates. Secrets created by cert-manager use the kubernetes.io/tls type and work with this configuration pattern.

Procedure

  1. Create a TLS secret that contains the client certificate and private key that the Telemetry Gateway uses toward the upstream collector: 
    $ oc create secret tls telemetry-gateway-forward-certs \
  --cert=/path/to/client-cert.crt \
  --key=/path/to/client-key.key \
  -n <project_name>
  2. Create a ConfigMap that contains the upstream certificate authority (CA) bundle: 
    $ oc create configmap telemetry-gateway-forward-ca \
  --from-file=ca.crt=/path/to/upstream-ca.crt \
  -n <project_name>
  3. Patch the flightctl-telemetry-gateway Deployment so the container can read the TLS materials at the paths referenced in the forwarding configuration. The chart templates do not mount arbitrary volumes for forwarding, so add volumes and volumeMounts that reference the secret and ConfigMap you created. Save the following as telemetry-gateway-forward-volumes.patch.yaml: 
    spec:
  template:
    spec:
      containers:
      - name: telemetry-gateway
        volumeMounts:
        - mountPath: /etc/telemetry-gateway/forward/certs
          name: forward-certs
          readOnly: true
        - mountPath: /etc/telemetry-gateway/forward/ca
          name: forward-ca
          readOnly: true
      volumes:
      - name: forward-certs
        secret:
          secretName: telemetry-gateway-forward-certs
      - name: forward-ca
        configMap:
          name: telemetry-gateway-forward-ca
     
    $ oc patch deployment flightctl-telemetry-gateway \
  --type strategic \
  --patch-file telemetry-gateway-forward-volumes.patch.yaml \
  -n <project_name>
  4. Edit the flightctl-telemetry-gateway-configConfigMap and add a forward block under the existing telemetryGateway key in config.yaml. Keep the existing listen and export keys; adjust endpoint and file paths to match your environment: 
    $ oc edit configmap flightctl-telemetry-gateway-config -n <project_name>

    Add the following under telemetryGateway in config.yaml:

          forward:
        endpoint: otlp.example.com:4317
        tls:
          insecureSkipTlsVerify: false
          caFile: /etc/telemetry-gateway/forward/ca/ca.crt
          certFile: /etc/telemetry-gateway/forward/certs/tls.crt
          keyFile: /etc/telemetry-gateway/forward/certs/tls.key
  5. Restart the Telemetry Gateway Deployment so it reloads the updated configuration and volume mounts: 
    $ oc rollout restart deployment/flightctl-telemetry-gateway -n <project_name>
$ oc rollout status deployment/flightctl-telemetry-gateway -n <project_name>
    Note

    A later helm upgrade re-renders the flightctl-telemetry-gateway-configConfigMap and the Telemetry Gateway Deployment from the chart templates. That action removes forwarding configuration and volume mounts added in this procedure. If forwarding stops working after an upgrade, repeat the ConfigMap edit, Deployment patch, and rollout restart steps (or merge equivalent changes by another supported method).

Results

Follow the Telemetry Gateway logs and confirm that batches are exported successfully to your OTLP endpoint. Log messages vary by version; consult your server and gateway logs if forwarding does not appear.