×

Distributed tracing overview

As a service owner, you can use distributed tracing to instrument your services to gather insights into your service architecture. You can use the Red Hat OpenShift distributed tracing platform for monitoring, network profiling, and troubleshooting the interaction between components in modern, cloud-native, microservices-based applications.

With the distributed tracing platform, you can perform the following functions:

  • Monitor distributed transactions

  • Optimize performance and latency

  • Perform root cause analysis

The distributed tracing platform consists of three components:

  • Red Hat OpenShift distributed tracing platform (Jaeger), which is based on the open source Jaeger project.

  • Red Hat OpenShift distributed tracing platform (Tempo), which is based on the open source Grafana Tempo project.

  • Red Hat build of OpenTelemetry, which is based on the open source OpenTelemetry project.

Component versions in the Red Hat OpenShift distributed tracing platform 2.9.2

Operator Component Version

Red Hat OpenShift distributed tracing platform (Jaeger)

Jaeger

1.47.0

Red Hat build of OpenTelemetry

OpenTelemetry

0.81.0

Red Hat OpenShift distributed tracing platform (Tempo)

Tempo

2.1.1

CVEs

This release fixes CVE-2023-46234.

Red Hat OpenShift distributed tracing platform (Jaeger)

Known issues

  • Apache Spark is not supported.

  • The streaming deployment via AMQ/Kafka is unsupported on IBM Z and IBM Power Systems.

Red Hat OpenShift distributed tracing platform (Tempo)

The Red Hat OpenShift distributed tracing platform (Tempo) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Known issues

  • Currently, the custom TLS CA option is not implemented for connecting to object storage. (TRACING-3462)

  • Currently, when used with the Tempo Operator, the Jaeger UI only displays services that have sent traces in the last 15 minutes. For services that did not send traces in the last 15 minutes, traces are still stored but not displayed in the Jaeger UI. (TRACING-3139)

  • Currently, the distributed tracing platform (Tempo) fails on the IBM Z (s390x) architecture. (TRACING-3545)

  • Currently, the Tempo query frontend service must not use internal mTLS when Gateway is not deployed. This issue does not affect the Jaeger Query API. The workaround is to disable mTLS. (TRACING-3510)

    Workaround

    Disable mTLS as follows:

    1. Open the Tempo Operator ConfigMap for editing by running the following command:

      $ oc edit configmap tempo-operator-manager-config -n openshift-tempo-operator (1)
      1 The project where the Tempo Operator is installed.
    2. Disable the mTLS in the operator configuration by updating the YAML file:

      data:
        controller_manager_config.yaml: |
          featureGates:
            httpEncryption: false
            grpcEncryption: false
            builtInCertManagement:
              enabled: false
    3. Restart the Tempo Operator pod by running the following command:

      $ oc rollout restart deployment.apps/tempo-operator-controller -n openshift-tempo-operator
  • Missing images for running the Tempo Operator in restricted environments. The Red Hat OpenShift distributed tracing platform (Tempo) CSV is missing references to the operand images. (TRACING-3523)

    Workaround

    Add the Tempo Operator related images in the mirroring tool to mirror the images to the registry:

    kind: ImageSetConfiguration
    apiVersion: mirror.openshift.io/v1alpha2
    archiveSize: 20
    storageConfig:
      local:
        path: /home/user/images
    mirror:
      operators:
      - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.13
        packages:
        - name: tempo-product
          channels:
          - name: stable
      additionalImages:
      - name: registry.redhat.io/rhosdt/tempo-rhel8@sha256:e4295f837066efb05bcc5897f31eb2bdbd81684a8c59d6f9498dd3590c62c12a
      - name: registry.redhat.io/rhosdt/tempo-gateway-rhel8@sha256:b62f5cedfeb5907b638f14ca6aaeea50f41642980a8a6f87b7061e88d90fac23
      - name: registry.redhat.io/rhosdt/tempo-gateway-opa-rhel8@sha256:8cd134deca47d6817b26566e272e6c3f75367653d589f5c90855c59b2fab01e9
      - name: registry.redhat.io/rhosdt/tempo-query-rhel8@sha256:0da43034f440b8258a48a0697ba643b5643d48b615cdb882ac7f4f1f80aad08e

Red Hat build of OpenTelemetry

The Red Hat build of OpenTelemetry is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Known issues

Getting support

If you experience difficulty with a procedure described in this documentation, or with OpenShift Container Platform in general, visit the Red Hat Customer Portal. From the Customer Portal, you can:

  • Search or browse through the Red Hat Knowledgebase of articles and solutions relating to Red Hat products.

  • Submit a support case to Red Hat Support.

  • Access other product documentation.

To identify issues with your cluster, you can use Insights in OpenShift Cluster Manager Hybrid Cloud Console. Insights provides details about issues and, if available, information on how to solve a problem.

If you have a suggestion for improving this documentation or have found an error, submit a Jira issue for the most relevant documentation component. Please provide specific details, such as the section name and OpenShift Container Platform version.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright’s message.