1. Documentation
    2. history bug_report picture_as_pdf
×

Configuring and deploying the distributed tracing platform (Tempo)

The Tempo Operator uses a custom resource definition (CRD) file that defines the architecture and configuration settings to be used when creating and deploying the distributed tracing platform (Tempo) resources. You can install the default configuration or modify the file.

Customizing your deployment

For information about configuring the back-end storage, see Understanding persistent storage and the appropriate configuration topic for your chosen storage option.

Distributed tracing default configuration options

The Tempo custom resource (CR) defines the architecture and settings to be used when creating the distributed tracing platform (Tempo) resources. You can modify these parameters to customize your distributed tracing platform (Tempo) implementation to your business needs.

Example of a generic Tempo YAML file
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: name
spec:
  storage: {}
  resources: {}
  storageSize: 200M
  replicationFactor: 1
  retention: {}
  template:
      distributor:{}
      ingester: {}
      compactor: {}
      querier: {}
      queryFrontend: {}
      gateway: {}
Table 1. Tempo parameters
Parameter Description Values Default value

apiVersion:

API version to use when creating the object.

tempotracing.io/v1

tempotracing.io/v1

kind:

Defines the kind of Kubernetes object to create.

tempo

metadata:

Data that uniquely identifies the object, including a name string, UID, and optional namespace.

OpenShift Container Platform automatically generates the UID and completes the namespace with the name of the project where the object is created.

name:

Name for the object.

Name of your TempoStack instance.

tempo-all-in-one-inmemory

spec:

Specification for the object to be created.

Contains all of the configuration parameters for your TempoStack instance. When a common definition for all Tempo components is required, it is defined under the spec node. When the definition relates to an individual component, it is placed under the spec/template/<component> node.

N/A

resources:

Resources assigned to the TempoStack.

storageSize:

Storage size for ingester PVCs.

replicationFactor:

Configuration for the replication factor.

retention:

Configuration options for retention of traces.

storage:

Configuration options that define the storage. All storage-related options must be placed under storage and not under the allInOne or other component options.

template.distributor:

Configuration options for the Tempo distributor.

template.ingester:

Configuration options for the Tempo ingester.

template.compactor:

Configuration options for the Tempo compactor.

template.querier:

Configuration options for the Tempo querier.

template.queryFrontend:

Configuration options for the Tempo query-frontend.

template.gateway:

Configuration options for the Tempo gateway.
Minimum required configuration

The following is the required minimum for creating a distributed tracing platform (Tempo) deployment with the default settings:

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  storage: (1)
    secret:
      name: minio
      type: s3
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
        ingress:
          type: route
1 This section specifies the deployed object storage back end, which requires a created secret with credentials for access to the object storage.

The distributed tracing platform (Tempo) storage configuration

You can configure object storage for the distributed tracing platform (Tempo) in the TempoStack custom resource under spec.storage. You can choose from among several storage providers that are supported.

Table 2. General storage parameters used by the Tempo Operator to define distributed tracing storage
Parameter Description Values Default value 
spec:
  storage:
    secret
      type:

Type of storage to use for the deployment.

memory. Memory storage is only appropriate for development, testing, demonstrations, and proof of concept environments because the data does not persist when the pod is shut down.

memory

 
storage:
  secretname:

Name of the secret that contains the credentials for the set object storage type.

N/A

 
storage:
  tls:
    caName:

CA is the name of a ConfigMap object containing a CA certificate.
Table 3. Required secret parameters
Storage provider Secret parameters

Red Hat OpenShift Data Foundation

name: tempostack-dev-odf # example

bucket: <bucket_name> # requires an ObjectBucketClaim

endpoint: https://s3.openshift-storage.svc

access_key_id: <data_foundation_access_key_id>

access_key_secret: <data_foundation_access_key_secret>

MinIO

See MinIO Operator.

name: tempostack-dev-minio # example

bucket: <minio_bucket_name> # MinIO documentation

endpoint: <minio_bucket_endpoint>

access_key_id: <minio_access_key_id>

access_key_secret: <minio_access_key_secret>

Amazon S3

name: tempostack-dev-s3 # example

bucket: <s3_bucket_name> # Amazon S3 documentation

endpoint: <s3_bucket_endpoint>

access_key_id: <s3_access_key_id>

access_key_secret: <s3_access_key_secret>

Microsoft Azure Blob Storage

name: tempostack-dev-azure # example

container: <azure_blob_storage_container_name> # Microsoft Azure documentation

account_name: <azure_blob_storage_account_name>

account_key: <azure_blob_storage_account_key>

Google Cloud Storage on Google Cloud Platform (GCP)

name: tempostack-dev-gcs # example

bucketname: <google_cloud_storage_bucket_name> # requires a bucket created in a GCP project

key.json: <path/to/key.json> # requires a service account in the bucket’s GCP project for GCP authentication

Query configuration options

Query is a service that retrieves traces from storage and hosts the user interface to display them.

Table 4. Parameters used by the Tempo Operator to define Query
Parameter Description Values Default value 
spec:
  query:
    replicas:

Specifies the number of Query replicas to create.

Positive integer
Table 5. Configuration parameters passed to Query
Parameter Description Values Default value 
spec:
  query:
    options: {}

Configuration options that define the Query service.

 
options:
  log-level:

Logging level for Query.

debug, info, warn, error, fatal, panic

 
options:
  query:
    base-path:

You can set the base path for all tempo-query HTTP routes to a non-root value: for example, /tempo will cause all UI URLs to start with /tempo. This can be useful when running tempo-query behind a reverse proxy.

/<path>
Sample Query configuration
apiVersion: tempotracing.io/v1
kind: "Tempo"
metadata:
  name: "my-tempo"
spec:
  strategy: allInOne
  allInOne:
    options:
      log-level: debug
      query:
        base-path: /tempo

Setting up monitoring for the distributed tracing platform (Tempo)

The Tempo Operator supports monitoring and alerting of each TempoStack component such as distributor, ingester, and so on, and exposes upgrade and operational metrics about the Operator itself.

Configuring TempoStack metrics and alerts

You can enable metrics and alerts of TempoStack instances.

Prerequisites
Procedure

  1. To enable metrics of a TempoStack instance, set the spec.observability.metrics.createServiceMonitors field to true:

    apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: <name>
spec:
  observability:
    metrics:
      createServiceMonitors: true

  2. To enable alerts for a TempoStack instance, set the spec.observability.metrics.createPrometheusRules field to true:

    apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: <name>
spec:
  observability:
    metrics:
      createPrometheusRules: true
Verification

You can use the Administrator view of the web console to verify successful configuration:

  1. Go to Observe → Targets, filter for Source: User, and check that ServiceMonitors in the format tempo-<instance_name>-<component> have the Up status.

  2. To verify that alerts are set up correctly, go to Observe → Alerting → Alerting rules, filter for Source: User, and check that the Alert rules for the TempoStack instance components are available.

Configuring Tempo Operator metrics and alerts

When installing the Tempo Operator from the web console, you can select the Enable Operator recommended cluster monitoring on this Namespace checkbox, which enables creating metrics and alerts of the Tempo Operator.

If the checkbox was not selected during installation, you can manually enable metrics and alerts even after installing the Tempo Operator.

Procedure

  • Add the openshift.io/cluster-monitoring: "true" label in the project where the Tempo Operator is installed, which is openshift-tempo-operator by default.

Verification

You can use the Administrator view of the web console to verify successful configuration:

  1. Go to Observe → Targets, filter for Source: Platform, and search for tempo-operator, which must have the Up status.

  2. To verify that alerts are set up correctly, go to Observe → Alerting → Alerting rules, filter for Source: Platform, and locate the Alert rules for the Tempo Operator.