1. Documentation
    2. history bug_report picture_as_pdf
×

Configuring and deploying distributed tracing

The Red Hat OpenShift distributed tracing platform Operator uses a custom resource definition (CRD) file that defines the architecture and configuration settings to be used when creating and deploying the distributed tracing platform resources. You can either install the default configuration or modify the file to better suit your business requirements.

Red Hat OpenShift distributed tracing platform has predefined deployment strategies. You specify a deployment strategy in the custom resource file. When you create a distributed tracing platform instance the Operator uses this configuration file to create the objects necessary for the deployment.

Jaeger custom resource file showing deployment strategy
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: MyConfigFile
spec:
  strategy: production (1)
1 The Red Hat OpenShift distributed tracing platform Operator currently supports the following deployment strategies:

  • allInOne (Default) - This strategy is intended for development, testing, and demo purposes; it is not intended for production use. The main backend components, Agent, Collector, and Query service, are all packaged into a single executable which is configured, by default. to use in-memory storage.

    In-memory storage is not persistent, which means that if the distributed tracing platform instance shuts down, restarts, or is replaced, that your trace data will be lost. And in-memory storage cannot be scaled, since each pod has its own memory. For persistent storage, you must use the production or streaming strategies, which use Elasticsearch as the default storage.

  • production - The production strategy is intended for production environments, where long term storage of trace data is important, as well as a more scalable and highly available architecture is required. Each of the backend components is therefore deployed separately. The Agent can be injected as a sidecar on the instrumented application. The Query and Collector services are configured with a supported storage type - currently Elasticsearch. Multiple instances of each of these components can be provisioned as required for performance and resilience purposes.

  • streaming - The streaming strategy is designed to augment the production strategy by providing a streaming capability that effectively sits between the Collector and the Elasticsearch backend storage. This provides the benefit of reducing the pressure on the backend storage, under high load situations, and enables other trace post-processing capabilities to tap into the real time span data directly from the streaming platform (AMQ Streams/ Kafka).

    The streaming strategy requires an additional Red Hat subscription for AMQ Streams.

The streaming deployment strategy is currently unsupported on IBM Z.

There are two ways to install and use Red Hat OpenShift distributed tracing, as part of a service mesh or as a stand alone component. If you have installed distributed tracing as part of Red Hat OpenShift Service Mesh, you can perform basic configuration as part of the ServiceMeshControlPlane but for completely control you should configure a Jaeger CR and then reference your distributed tracing configuration file in the ServiceMeshControlPlane.

Deploying the distributed tracing default strategy from the web console

The custom resource definition (CRD) defines the configuration used when you deploy an instance of Red Hat OpenShift distributed tracing. The default CR is named jaeger-all-in-one-inmemory and it is configured with minimal resources to ensure that you can successfully install it on a default OpenShift Container Platform installation. You can use this default configuration to create a Red Hat OpenShift distributed tracing platform instance that uses the AllInOne deployment strategy, or you can define your own custom resource file.

In-memory storage is not persistent. If the Jaeger pod shuts down, restarts, or is replaced, your trace data will be lost. For persistent storage, you must use the production or streaming strategies, which use Elasticsearch as the default storage.
Prerequisites

  • The Red Hat OpenShift distributed tracing platform Operator has been installed.

  • You have reviewed the instructions for how to customize the deployment.

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.

  2. Create a new project, for example tracing-system.

    If you are installing as part of Service Mesh, the distributed tracing resources must be installed in the same namespace as the ServiceMeshControlPlane resource, for example istio-system.

    1. Navigate to Home → Projects.

    2. Click Create Project.

    3. Enter tracing-system in the Name field.

    4. Click Create.

  3. Navigate to Operators → Installed Operators.

  4. If necessary, select tracing-system from the Project menu. You may have to wait a few moments for the Operators to be copied to the new project.

  5. Click the Red Hat OpenShift distributed tracing platform Operator. On the Details tab, under Provided APIs, the Operator provides a single link.

  6. Under Jaeger, click Create Instance.

  7. On the Create Jaeger page, to install using the defaults, click Create to create the distributed tracing platform instance.

  8. On the Jaegers page, click the name of the distributed tracing platform instance, for example, jaeger-all-in-one-inmemory.

  9. On the Jaeger Details page, click the Resources tab. Wait until the pod has a status of "Running" before continuing.

Deploying the distributed tracing default strategy from the CLI

Follow this procedure to create an instance of distributed tracing platform from the command line.

Prerequisites

  • The Red Hat OpenShift distributed tracing platform Operator has been installed and verified.

  • You have reviewed the instructions for how to customize the deployment.

  • You have access to the OpenShift CLI (oc) that matches your OpenShift Container Platform version.

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

    $ oc login --username=<NAMEOFUSER> https://<HOSTNAME>:8443

  2. Create a new project named tracing-system.

    $ oc new-project tracing-system

  3. Create a custom resource file named jaeger.yaml that contains the following text:

    Example jaeger-all-in-one.yaml
    apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-all-in-one-inmemory

  4. Run the following command to deploy distributed tracing platform:

    $ oc create -n tracing-system -f jaeger.yaml

  5. Run the following command to watch the progress of the pods during the installation process:

    $ oc get pods -n tracing-system -w

    After the installation process has completed, you should see output similar to the following example:

    NAME                                         READY   STATUS    RESTARTS   AGE
jaeger-all-in-one-inmemory-cdff7897b-qhfdx   2/2     Running   0          24s

Deploying the distributed tracing production strategy from the web console

The production deployment strategy is intended for production environments that require a more scalable and highly available architecture, and where long-term storage of trace data is important.

Prerequisites

  • The OpenShift Elasticsearch Operator has been installed.

  • The Red Hat OpenShift distributed tracing platform Operator has been installed.

  • You have reviewed the instructions for how to customize the deployment.

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Log in to the OpenShift Container Platform web console as a user with the cluster-admin role.

  2. Create a new project, for example tracing-system.

    If you are installing as part of Service Mesh, the distributed tracing resources must be installed in the same namespace as the ServiceMeshControlPlane resource, for example istio-system.

    1. Navigate to Home → Projects.

    2. Click Create Project.

    3. Enter tracing-system in the Name field.

    4. Click Create.

  3. Navigate to Operators → Installed Operators.

  4. If necessary, select tracing-system from the Project menu. You may have to wait a few moments for the Operators to be copied to the new project.

  5. Click the Red Hat OpenShift distributed tracing platform Operator. On the Overview tab, under Provided APIs, the Operator provides a single link.

  6. Under Jaeger, click Create Instance.

  7. On the Create Jaeger page, replace the default all-in-one YAML text with your production YAML configuration, for example:

    Example jaeger-production.yaml file with Elasticsearch
    apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-production
  namespace:
spec:
  strategy: production
  ingress:
    security: oauth-proxy
  storage:
    type: elasticsearch
    elasticsearch:
      nodeCount: 3
      redundancyPolicy: SingleRedundancy
    esIndexCleaner:
      enabled: true
      numberOfDays: 7
      schedule: 55 23 * * *
    esRollover:
      schedule: '*/30 * * * *'

  8. Click Create to create the distributed tracing platform instance.

  9. On the Jaegers page, click the name of the distributed tracing platform instance, for example, jaeger-prod-elasticsearch.

  10. On the Jaeger Details page, click the Resources tab. Wait until all the pods have a status of "Running" before continuing.

Deploying the distributed tracing production strategy from the CLI

Follow this procedure to create an instance of distributed tracing platform from the command line.

Prerequisites

  • The OpenShift Elasticsearch Operator has been installed.

  • The Red Hat OpenShift distributed tracing platform Operator has been installed.

  • You have reviewed the instructions for how to customize the deployment.

  • You have access to the OpenShift CLI (oc) that matches your OpenShift Container Platform version.

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  1. Log in to the OpenShift Container Platform CLI as a user with the cluster-admin role.

    $ oc login --username=<NAMEOFUSER> https://<HOSTNAME>:8443

  2. Create a new project named tracing-system.

    $ oc new-project tracing-system

  3. Create a custom resource file named jaeger-production.yaml that contains the text of the example file in the previous procedure.

  4. Run the following command to deploy distributed tracing platform:

    $ oc create -n tracing-system -f jaeger-production.yaml

  5. Run the following command to watch the progress of the pods during the installation process:

    $ oc get pods -n tracing-system -w

    After the installation process has completed, you should see output similar to the following example:

    NAME                                                              READY   STATUS    RESTARTS   AGE
elasticsearch-cdm-jaegersystemjaegerproduction-1-6676cf568gwhlw   2/2     Running   0          10m
elasticsearch-cdm-jaegersystemjaegerproduction-2-bcd4c8bf5l6g6w   2/2     Running   0          10m
elasticsearch-cdm-jaegersystemjaegerproduction-3-844d6d9694hhst   2/2     Running   0          10m
jaeger-production-collector-94cd847d-jwjlj                        1/1     Running   3          8m32s
jaeger-production-query-5cbfbd499d-tv8zf                          3/3     Running   3          8m32s

Deploying the distributed tracing streaming strategy from the web console

The streaming deployment strategy is intended for production environments that require a more scalable and highly available architecture, and where long-term storage of trace data is important.

The streaming strategy provides a streaming capability that sits between the Collector and the Elasticsearch storage. This reduces the pressure on the storage under high load situations, and enables other trace post-processing capabilities to tap into the real-time span data directly from the Kafka streaming platform.