apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: MyConfigFile
spec:
strategy: production (1)
The Red Hat OpenShift distributed tracing platform Operator uses a custom resource definition (CRD) file that defines the architecture and configuration settings to be used when creating and deploying the distributed tracing platform resources. You can either install the default configuration or modify the file to better suit your business requirements.
Red Hat OpenShift distributed tracing platform has predefined deployment strategies. You specify a deployment strategy in the custom resource file. When you create a distributed tracing platform instance the Operator uses this configuration file to create the objects necessary for the deployment.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: MyConfigFile
spec:
strategy: production (1)
1 | The Red Hat OpenShift distributed tracing platform Operator currently supports the following deployment strategies:
|
The streaming deployment strategy is currently unsupported on IBM Z. |
There are two ways to install and use Red Hat OpenShift distributed tracing, as part of a service mesh or as a stand alone component. If you have installed distributed tracing as part of Red Hat OpenShift Service Mesh, you can perform basic configuration as part of the ServiceMeshControlPlane but for completely control you should configure a Jaeger CR and then reference your distributed tracing configuration file in the ServiceMeshControlPlane. |
The custom resource definition (CRD) defines the configuration used when you deploy an instance of Red Hat OpenShift distributed tracing. The default CR is named jaeger-all-in-one-inmemory
and it is configured with minimal resources to ensure that you can successfully install it on a default OpenShift Container Platform installation. You can use this default configuration to create a Red Hat OpenShift distributed tracing platform instance that uses the AllInOne
deployment strategy, or you can define your own custom resource file.
In-memory storage is not persistent. If the Jaeger pod shuts down, restarts, or is replaced, your trace data will be lost. For persistent storage, you must use the |
The Red Hat OpenShift distributed tracing platform Operator has been installed.
You have reviewed the instructions for how to customize the deployment.
You have access to the cluster as a user with the cluster-admin
role.
Log in to the OpenShift Container Platform web console as a user with the cluster-admin
role.
Create a new project, for example tracing-system
.
If you are installing as part of Service Mesh, the distributed tracing resources must be installed in the same namespace as the |
Navigate to Home → Projects.
Click Create Project.
Enter tracing-system
in the Name field.
Click Create.
Navigate to Operators → Installed Operators.
If necessary, select tracing-system
from the Project menu. You may have to wait a few moments for the Operators to be copied to the new project.
Click the Red Hat OpenShift distributed tracing platform Operator. On the Details tab, under Provided APIs, the Operator provides a single link.
Under Jaeger, click Create Instance.
On the Create Jaeger page, to install using the defaults, click Create to create the distributed tracing platform instance.
On the Jaegers page, click the name of the distributed tracing platform instance, for example, jaeger-all-in-one-inmemory
.
On the Jaeger Details page, click the Resources tab. Wait until the pod has a status of "Running" before continuing.
Follow this procedure to create an instance of distributed tracing platform from the command line.
The Red Hat OpenShift distributed tracing platform Operator has been installed and verified.
You have reviewed the instructions for how to customize the deployment.
You have access to the OpenShift CLI (oc
) that matches your OpenShift Container Platform version.
You have access to the cluster as a user with the cluster-admin
role.
Log in to the OpenShift Container Platform CLI as a user with the cluster-admin
role.
$ oc login https://<HOSTNAME>:8443
Create a new project named tracing-system
.
$ oc new-project tracing-system
Create a custom resource file named jaeger.yaml
that contains the following text:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: jaeger-all-in-one-inmemory
Run the following command to deploy distributed tracing platform:
$ oc create -n tracing-system -f jaeger.yaml
Run the following command to watch the progress of the pods during the installation process:
$ oc get pods -n tracing-system -w
After the installation process has completed, you should see output similar to the following example:
NAME READY STATUS RESTARTS AGE
jaeger-all-in-one-inmemory-cdff7897b-qhfdx 2/2 Running 0 24s
The production
deployment strategy is intended for production environments that require a more scalable and highly available architecture, and where long-term storage of trace data is important.
The OpenShift Elasticsearch Operator has been installed.
The Red Hat OpenShift distributed tracing platform Operator has been installed.
You have reviewed the instructions for how to customize the deployment.
You have access to the cluster as a user with the cluster-admin
role.
Log in to the OpenShift Container Platform web console as a user with the cluster-admin
role.
Create a new project, for example tracing-system
.
If you are installing as part of Service Mesh, the distributed tracing resources must be installed in the same namespace as the |
Navigate to Home → Projects.
Click Create Project.
Enter tracing-system
in the Name field.
Click Create.
Navigate to Operators → Installed Operators.
If necessary, select tracing-system
from the Project menu. You may have to wait a few moments for the Operators to be copied to the new project.
Click the Red Hat OpenShift distributed tracing platform Operator. On the Overview tab, under Provided APIs, the Operator provides a single link.
Under Jaeger, click Create Instance.
On the Create Jaeger page, replace the default all-in-one
YAML text with your production YAML configuration, for example:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: jaeger-production
namespace:
spec:
strategy: production
ingress:
security: oauth-proxy
storage:
type: elasticsearch
elasticsearch:
nodeCount: 3
redundancyPolicy: SingleRedundancy
esIndexCleaner:
enabled: true
numberOfDays: 7
schedule: 55 23 * * *
esRollover:
schedule: '*/30 * * * *'
Click Create to create the distributed tracing platform instance.
On the Jaegers page, click the name of the distributed tracing platform instance, for example, jaeger-prod-elasticsearch
.
On the Jaeger Details page, click the Resources tab. Wait until all the pods have a status of "Running" before continuing.
Follow this procedure to create an instance of distributed tracing platform from the command line.
The OpenShift Elasticsearch Operator has been installed.
The Red Hat OpenShift distributed tracing platform Operator has been installed.
You have reviewed the instructions for how to customize the deployment.
You have access to the OpenShift CLI (oc
) that matches your OpenShift Container Platform version.
You have access to the cluster as a user with the cluster-admin
role.
Log in to the OpenShift Container Platform CLI as a user with the cluster-admin
role.
$ oc login https://<HOSTNAME>:8443
Create a new project named tracing-system
.
$ oc new-project tracing-system
Create a custom resource file named jaeger-production.yaml
that contains the text of the example file in the previous procedure.
Run the following command to deploy distributed tracing platform:
$ oc create -n tracing-system -f jaeger-production.yaml
Run the following command to watch the progress of the pods during the installation process:
$ oc get pods -n tracing-system -w
After the installation process has completed, you should see output similar to the following example:
NAME READY STATUS RESTARTS AGE
elasticsearch-cdm-jaegersystemjaegerproduction-1-6676cf568gwhlw 2/2 Running 0 10m
elasticsearch-cdm-jaegersystemjaegerproduction-2-bcd4c8bf5l6g6w 2/2 Running 0 10m
elasticsearch-cdm-jaegersystemjaegerproduction-3-844d6d9694hhst 2/2 Running 0 10m
jaeger-production-collector-94cd847d-jwjlj 1/1 Running 3 8m32s
jaeger-production-query-5cbfbd499d-tv8zf 3/3 Running 3 8m32s
The streaming
deployment strategy is intended for production environments that require a more scalable and highly available architecture, and where long-term storage of trace data is important.
The streaming
strategy provides a streaming capability that sits between the Collector and the Elasticsearch storage. This reduces the pressure on the storage under high load situations, and enables other trace post-processing capabilities to tap into the real-time span data directly from the Kafka streaming platform.
The streaming strategy requires an additional Red Hat subscription for AMQ Streams. If you do not have an AMQ Streams subscription, contact your sales representative for more information. |
The streaming deployment strategy is currently unsupported on IBM Z. |
The AMQ Streams Operator has been installed. If using version 1.4.0 or higher you can use self-provisioning. Otherwise you must create the Kafka instance.
The Red Hat OpenShift distributed tracing platform Operator has been installed.
You have reviewed the instructions for how to customize the deployment.
You have access to the cluster as a user with the cluster-admin
role.
Log in to the OpenShift Container Platform web console as a user with the cluster-admin
role.
Create a new project, for example tracing-system
.
If you are installing as part of Service Mesh, the distributed tracing resources must be installed in the same namespace as the |
Navigate to Home → Projects.
Click Create Project.
Enter tracing-system
in the Name field.
Click Create.
Navigate to Operators → Installed Operators.
If necessary, select tracing-system
from the Project menu. You may have to wait a few moments for the Operators to be copied to the new project.
Click the Red Hat OpenShift distributed tracing platform Operator. On the Overview tab, under Provided APIs, the Operator provides a single link.
Under Jaeger, click Create Instance.
On the Create Jaeger page, replace the default all-in-one
YAML text with your streaming YAML configuration, for example:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: jaeger-streaming
spec:
strategy: streaming
collector:
options:
kafka:
producer:
topic: jaeger-spans
#Note: If brokers are not defined,AMQStreams 1.4.0+ will self-provision Kafka.
brokers: my-cluster-kafka-brokers.kafka:9092
storage:
type: elasticsearch
ingester:
options:
kafka:
consumer:
topic: jaeger-spans
brokers: my-cluster-kafka-brokers.kafka:9092
Click Create to create the distributed tracing platform instance.
On the Jaegers page, click the name of the distributed tracing platform instance, for example, jaeger-streaming
.
On the Jaeger Details page, click the Resources tab. Wait until all the pods have a status of "Running" before continuing.
Follow this procedure to create an instance of distributed tracing platform from the command line.
The AMQ Streams Operator has been installed. If using version 1.4.0 or higher you can use self-provisioning. Otherwise you must create the Kafka instance.
The Red Hat OpenShift distributed tracing platform Operator has been installed.
You have reviewed the instructions for how to customize the deployment.
You have access to the OpenShift CLI (oc
) that matches your OpenShift Container Platform version.
You have access to the cluster as a user with the cluster-admin
role.
Procedure
Log in to the OpenShift Container Platform CLI as a user with the cluster-admin
role.
$ oc login https://<HOSTNAME>:8443
Create a new project named tracing-system
.
$ oc new-project tracing-system
Create a custom resource file named jaeger-streaming.yaml
that contains the text of the example file in the previous procedure.
Run the following command to deploy Jaeger:
$ oc create -n tracing-system -f jaeger-streaming.yaml
Run the following command to watch the progress of the pods during the installation process:
$ oc get pods -n tracing-system -w
After the installation process has completed, you should see output similar to the following example:
NAME READY STATUS RESTARTS AGE
elasticsearch-cdm-jaegersystemjaegerstreaming-1-697b66d6fcztcnn 2/2 Running 0 5m40s
elasticsearch-cdm-jaegersystemjaegerstreaming-2-5f4b95c78b9gckz 2/2 Running 0 5m37s
elasticsearch-cdm-jaegersystemjaegerstreaming-3-7b6d964576nnz97 2/2 Running 0 5m5s
jaeger-streaming-collector-6f6db7f99f-rtcfm 1/1 Running 0 80s
jaeger-streaming-entity-operator-6b6d67cc99-4lm9q 3/3 Running 2 2m18s
jaeger-streaming-ingester-7d479847f8-5h8kc 1/1 Running 0 80s
jaeger-streaming-kafka-0 2/2 Running 0 3m1s
jaeger-streaming-query-65bf5bb854-ncnc7 3/3 Running 0 80s
jaeger-streaming-zookeeper-0 2/2 Running 0 3m39s
To access the Jaeger console you must have either Red Hat OpenShift Service Mesh or Red Hat OpenShift distributed tracing installed, and Red Hat OpenShift distributed tracing platform installed, configured, and deployed.
The installation process creates a route to access the Jaeger console.
If you know the URL for the Jaeger console, you can access it directly. If you do not know the URL, use the following directions.
Log in to the OpenShift Container Platform web console as a user with cluster-admin rights. If you use Red Hat OpenShift Dedicated, you must have an account with the dedicated-admin
role.
Navigate to Networking → Routes.
On the Routes page, select the control plane project, for example tracing-system
, from the Namespace menu.
The Location column displays the linked address for each route.
If necessary, use the filter to find the jaeger
route. Click the route Location to launch the console.
Click Log In With OpenShift.
Log in to the OpenShift Container Platform CLI as a user with the cluster-admin
role. If you use Red Hat OpenShift Dedicated, you must have an account with the dedicated-admin
role.
$ oc login https://<HOSTNAME>:6443
To query for details of the route using the command line, enter the following command. In this example, tracing-system
is the control plane namespace.
$ export JAEGER_URL=$(oc get route -n tracing-system jaeger -o jsonpath='{.spec.host}')
Launch a browser and navigate to https://<JAEGER_URL>
, where <JAEGER_URL>
is the route that you discovered in the previous step.
Log in using the same user name and password that you use to access the OpenShift Container Platform console.
If you have added services to the service mesh and have generated traces, you can use the filters and Find Traces button to search your trace data.
If you are validating the console installation, there is no trace data to display.
Red Hat OpenShift distributed tracing instance names must be unique. If you want to have multiple Red Hat OpenShift distributed tracing platform instances and are using sidecar injected agents, then the Red Hat OpenShift distributed tracing platform instances should have unique names, and the injection annotation should explicitly specify the Red Hat OpenShift distributed tracing platform instance name the tracing data should be reported to.
If you have a multitenant implementation and tenants are separated by namespaces, deploy a Red Hat OpenShift distributed tracing platform instance to each tenant namespace.
Agent as a daemonset is not supported for multitenant installations or Red Hat OpenShift Dedicated. Agent as a sidecar is the only supported configuration for these use cases.
If you are installing distributed tracing as part of Red Hat OpenShift Service Mesh, the distributed tracing resources must be installed in the same namespace as the ServiceMeshControlPlane
resource.
The Jaeger custom resource (CR) defines the architecture and settings to be used when creating the distributed tracing platform resources. You can modify these parameters to customize your distributed tracing platform implementation to your business needs.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: name
spec:
strategy: <deployment_strategy>
allInOne:
options: {}
resources: {}
agent:
options: {}
resources: {}
collector:
options: {}
resources: {}
sampling:
options: {}
storage:
type:
options: {}
query:
options: {}
resources: {}
ingester:
options: {}
resources: {}
options: {}
Parameter | Description | Values | Default value |
---|---|---|---|
|
API version to use when creating the object. |
|
|
|
|
Defines the kind of Kubernetes object to create. |
|
|
Data that helps uniquely identify the object, including a |
||
OpenShift Container Platform automatically generates the |
|
Name for the object. |
The name of your distributed tracing platform instance. |
|
|
Specification for the object to be created. |
Contains all of the configuration parameters for your distributed tracing platform instance. When a common definition for all Jaeger components is required, it is defined under the |
N/A |
|
Jaeger deployment strategy |
|
|
|
Because the |
|
|
Configuration options that define the Agent. |
||
|
Configuration options that define the Jaeger Collector. |
||
|
Configuration options that define the sampling strategies for tracing. |
||
|
Configuration options that define the storage. All storage-related options must be placed under |
||
|
Configuration options that define the Query service. |
||
|
Configuration options that define the Ingester service. |
The following example YAML is the minimum required to create a Red Hat OpenShift distributed tracing platform deployment using the default settings.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: jaeger-all-in-one-inmemory
The Jaeger Collector is the component responsible for receiving the spans that were captured by the tracer and writing them to persistent Elasticsearch storage when using the production
strategy, or to AMQ Streams when using the streaming
strategy.
The Collectors are stateless and thus many instances of Jaeger Collector can be run in parallel. Collectors require almost no configuration, except for the location of the Elasticsearch cluster.
Parameter | Description | Values |
---|---|---|
collector: replicas: |
Specifies the number of Collector replicas to create. |
Integer, for example, |
Parameter | Description | Values |
---|---|---|
spec: collector: options: {} |
Configuration options that define the Jaeger Collector. |
|
options: collector: num-workers: |
The number of workers pulling from the queue. |
Integer, for example, |
options: collector: queue-size: |
The size of the Collector queue. |
Integer, for example, |
options: kafka: producer: topic: jaeger-spans |
The |
Label for the producer. |
options: kafka: producer: brokers: my-cluster-kafka-brokers.kafka:9092 |
Identifies the Kafka configuration used by the Collector to produce the messages. If brokers are not specified, and you have AMQ Streams 1.4.0+ installed, the Red Hat OpenShift distributed tracing platform Operator will self-provision Kafka. |
|
options: log-level: |
Logging level for the Collector. |
Possible values: |
The Red Hat OpenShift distributed tracing platform Operator can be used to define sampling strategies that will be supplied to tracers that have been configured to use a remote sampler.
While all traces are generated, only a few are sampled. Sampling a trace marks the trace for further processing and storage.
This is not relevant if a trace was started by the Envoy proxy, as the sampling decision is made there. The Jaeger sampling decision is only relevant when the trace is started by an application using the client. |
When a service receives a request that contains no trace context, the client starts a new trace, assigns it a random trace ID, and makes a sampling decision based on the currently installed sampling strategy. The sampling decision propagates to all subsequent requests in the trace so that other services are not making the sampling decision again.
distributed tracing platform libraries support the following samplers:
Probabilistic - The sampler makes a random sampling decision with the probability of sampling equal to the value of the sampling.param
property. For example, using sampling.param=0.1
samples approximately 1 in 10 traces.
Rate Limiting - The sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, using sampling.param=2.0
samples requests with the rate of 2 traces per second.
Parameter | Description | Values | Default value |
---|---|---|---|
spec: sampling: options: {} default_strategy: service_strategy: |
Configuration options that define the sampling strategies for tracing. |
If you do not provide configuration, the Collectors will return the default probabilistic sampling policy with 0.001 (0.1%) probability for all services. |
|
default_strategy: type: service_strategy: type: |
Sampling strategy to use. See descriptions above. |
Valid values are |
|
default_strategy: param: service_strategy: param: |
Parameters for the selected sampling strategy. |
Decimal and integer values (0, .1, 1, 10) |
1 |
This example defines a default sampling strategy that is probabilistic, with a 50% chance of the trace instances being sampled.
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: with-sampling
spec:
sampling:
options:
default_strategy:
type: probabilistic
param: 0.5
service_strategies:
- service: alpha
type: probabilistic
param: 0.8
operation_strategies:
- operation: op1
type: probabilistic
param: 0.2
- operation: op2
type: probabilistic
param: 0.4
- service: beta
type: ratelimiting
param: 5
If there are no user-supplied configurations, the distributed tracing platform uses the following settings:
spec:
sampling:
options:
default_strategy:
type: probabilistic
param: 1
You configure storage for the Collector, Ingester, and Query services under spec.storage
. Multiple instances of each of these components can be provisioned as required for performance and resilience purposes.
Parameter | Description | Values | Default value |
---|---|---|---|
spec: storage: type: |
Type of storage to use for the deployment. |
|
|
storage: secretname: |
Name of the secret, for example |
N/A |
|
storage: options: {} |
Configuration options that define the storage. |
Parameter | Description | Values | Default value |
---|---|---|---|
storage: esIndexCleaner: enabled: |
When using Elasticsearch storage, by default a job is created to clean old traces from the index. This parameter enables or disables the index cleaner job. |
|
|
storage: esIndexCleaner: numberOfDays: |
Number of days to wait before deleting an index. |
Integer value |
|
storage: esIndexCleaner: schedule: |
Defines the schedule for how often to clean the Elasticsearch index. |
Cron expression |
"55 23 * * *" |
When you deploy a Jaeger custom resource, the Red Hat OpenShift distributed tracing platform Operator uses the OpenShift Elasticsearch Operator to create an Elasticsearch cluster based on the configuration provided in the storage
section of the custom resource file. The Red Hat OpenShift distributed tracing platform Operator will provision Elasticsearch if the following configurations are set:
spec.storage:type
is set to elasticsearch
spec.storage.elasticsearch.doNotProvision
set to false
spec.storage.options.es.server-urls
is not defined, that is, there is no connection to an Elasticsearch instance that was not provisioned by the Red Hat Elasticsearch Operator.
When provisioning Elasticsearch, the Red Hat OpenShift distributed tracing platform Operator sets the Elasticsearch custom resource name
to the value of spec.storage.elasticsearch.name
from the Jaeger custom resource. If you do not specify a value for spec.storage.elasticsearch.name
, the Operator uses elasticsearch
.
You can have only one distributed tracing platform with self-provisioned Elasticsearch instance per namespace. The Elasticsearch cluster is meant to be dedicated for a single distributed tracing platform instance.
There can be only one Elasticsearch per namespace.
If you already have installed Elasticsearch as part of OpenShift Logging, the Red Hat OpenShift distributed tracing platform Operator can use the installed OpenShift Elasticsearch Operator to provision storage. |
The following configuration parameters are for a self-provisioned Elasticsearch instance, that is an instance created by the Red Hat OpenShift distributed tracing platform Operator using the OpenShift Elasticsearch Operator. You specify configuration options for self-provisioned Elasticsearch under spec:storage:elasticsearch
in your configuration file.
Parameter | Description | Values | Default value |
---|---|---|---|
elasticsearch: properties: doNotProvision: |
Use to specify whether or not an Elasticsearch instance should be provisioned by the Red Hat OpenShift distributed tracing platform Operator. |
|
|
elasticsearch: properties: name: |
Name of the Elasticsearch instance. The Red Hat OpenShift distributed tracing platform Operator uses the Elasticsearch instance specified in this parameter to connect to Elasticsearch. |
string |
|
elasticsearch: nodeCount: |
Number of Elasticsearch nodes. For high availability use at least 3 nodes. Do not use 2 nodes as “split brain” problem can happen. |
Integer value. For example, Proof of concept = 1, Minimum deployment =3 |
3 |
elasticsearch: resources: requests: cpu: |
Number of central processing units for requests, based on your environment’s configuration. |
Specified in cores or millicores, for example, 200m, 0.5, 1. For example, Proof of concept = 500m, Minimum deployment =1 |
1 |
elasticsearch: resources: requests: memory: |
Available memory for requests, based on your environment’s configuration. |
Specified in bytes, for example, 200Ki, 50Mi, 5Gi. For example, Proof of concept = 1Gi, Minimum deployment = 16Gi* |
16Gi |
elasticsearch: resources: limits: cpu: |
Limit on number of central processing units, based on your environment’s configuration. |
Specified in cores or millicores, for example, 200m, 0.5, 1. For example, Proof of concept = 500m, Minimum deployment =1 |