1. Documentation
    2. history bug_report picture_as_pdf
×

Configuring and deploying the distributed tracing data collection

The Red Hat OpenShift distributed tracing data collection Operator uses a custom resource definition (CRD) file that defines the architecture and configuration settings to be used when creating and deploying the distributed tracing data collection resources. You can install the default configuration or modify the file.

OpenTelemetry Collector configuration options

The OpenTelemetry Collector consists of three components that access telemetry data:

Receivers

A receiver, which can be push or pull based, is how data gets into the Collector. Generally, a receiver accepts data in a specified format, translates it into the internal format, and passes it to processors and exporters defined in the applicable pipelines. By default, no receivers are configured. One or more receivers must be configured. Receivers may support one or more data sources.

Processors

Optional. Processors run through the data between it is received and exported. By default, no processors are enabled. Processors must be enabled for every data source. Not all processors support all data sources. Depending on the data source, multiple processors might be enabled. Note that the order of processors matters.

Exporters

An exporter, which can be push or pull based, is how you send data to one or more back ends or destinations. By default, no exporters are configured. One or more exporters must be configured. Exporters can support one or more data sources. Exporters might be used with their default settings, but many exporters require configuration to specify at least the destination and security settings.

You can define multiple instances of components in a custom resource YAML file. When configured, these components must be enabled through pipelines defined in the spec.config.service section of the YAML file. As a best practice, only enable the components that you need.

Example of the OpenTelemetry Collector custom resource file
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: cluster-collector
  namespace: tracing-system
spec:
  mode: deployment
  ports:
  - name: promexporter
    port: 8889
    protocol: TCP
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
    exporters:
      jaeger:
        endpoint: jaeger-production-collector-headless.tracing-system.svc:14250
        tls:
          ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"
      prometheus:
        endpoint: 0.0.0.0:8889
        resource_to_telemetry_conversion:
          enabled: true # by default resource attributes are dropped
    service: (1)
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [jaeger]
        metrics:
          receivers: [otlp]
          processors: []
          exporters: [prometheus]
1 If a component is configured but not defined in the service section, the component is not enabled.
Table 1. Parameters used by the Operator to define the OpenTelemetry Collector
Parameter Description Values Default 
receivers:

A receiver is how data gets into the Collector. By default, no receivers are configured. There must be at least one enabled receiver for a configuration to be considered valid. Receivers are enabled by being added to a pipeline.

otlp, jaeger, zipkin

None

 
processors:

Processors run through the data between it is received and exported. By default, no processors are enabled.

None

 
exporters:

An exporter sends data to one or more back ends or destinations. By default, no exporters are configured. There must be at least one enabled exporter for a configuration to be considered valid. Exporters are enabled by being added to a pipeline. Exporters might be used with their default settings, but many require configuration to specify at least the destination and security settings.

otlp, otlphttp, jaeger, logging, prometheus

None

 
service:
  pipelines:

Components are enabled by adding them to a pipeline under services.pipeline.

 
service:
  pipelines:
    traces:
      receivers:

You enable receivers for tracing by adding them under service.pipelines.traces.

None

 
service:
  pipelines:
    traces:
      processors:

You enable processors for tracing by adding them under service.pipelines.traces.

None

 
service:
  pipelines:
    traces:
      exporters:

You enable exporters for tracing by adding them under service.pipelines.traces.

None

 
service:
  pipelines:
    metrics:
      receivers:

You enable receivers for metrics by adding them under service.pipelines.metrics.

None

 
service:
  pipelines:
    metrics:
      processors:

You enable processors for metircs by adding them under service.pipelines.metrics.

None

 
service:
  pipelines:
    metrics:
      exporters:

You enable exporters for metrics by adding them under service.pipelines.metrics.

None

OpenTelemetry Collector components

Receivers

OTLP Receiver

The OTLP receiver ingests data using the OpenTelemetry protocol (OTLP).

OpenTelemetry Collector custom resource with an enabled OTLP receiver
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317 (1)
          http:
            endpoint: 0.0.0.0:4318 (2)
          tls: (3)
            ca_file: ca.pem
            cert_file: cert.pem
            key_file: key.pem

    service:
      pipelines:
        traces:
          receivers: [otlp]
        metrics:
          receivers: [otlp]
1 The OTLP gRPC endpoint. If omitted, the default 0.0.0.0:4317 is used.
2 The OTLP HTTP endpoint. If omitted, the default 0.0.0.0:4318 is used.
3 The TLS server side configuration. Defines paths to TLS certificates. If omitted, TLS is disabled.
Jaeger Receiver

The Jaeger receiver ingests data in Jaeger formats.

OpenTelemetry Collector custom resource with an enabled Jaeger receiver
  config: |
    receivers:
      jaeger:
        protocols:
          grpc:
            endpoint: 0.0.0.0:14250 (1)
          thrift_http:
            endpoint: 0.0.0.0:14268 (2)
          thrift_compact:
            endpoint: 0.0.0.0:6831 (3)
          thrift_binary:
            endpoint: 0.0.0.0:6832 (4)
          tls: (5)

    service:
      pipelines:
        traces:
          receivers: [jaeger]
1 The Jaeger gRPC endpoint. If omitted, the default 0.0.0.0:14250 is used.
2 The Jaeger Thrift HTTP endpoint. If omitted, the default 0.0.0.0:14268 is used.
3 The Jaeger Thrift Compact endpoint. If omitted, the default 0.0.0.0:6831 is used.
4 The Jaeger Thrift Binary endpoint. If omitted, the default 0.0.0.0:6832 is used.
5 The TLS server side configuration. See the OTLP receiver configuration section for more details.
Zipkin Receiver

The Zipkin receiver ingests data in the Zipkin v1 and v2 formats.

OpenTelemetry Collector custom resource with enabled Zipkin receiver
  config: |
    receivers:
      zipkin:
        endpoint: 0.0.0.0:9411 (1)
        tls: (2)

    service:
      pipelines:
        traces:
          receivers: [zipkin]
1 The Zipkin HTTP endpoint. If omitted, the default 0.0.0.0:9411 is used.
2 The TLS server side configuration. See the OTLP receiver configuration section for more details.

Processors

Resource Detection processor

The Resource Detection processor is designed to identify host resource details in alignment with OpenTelemetry’s resource semantic standards. Using this detected information, it can add or replace the resource values in telemetry data.

OpenShift Container Platform permissions required for the Resource Detection processor
kind: ClusterRole
metadata:
  name: otel-collector
rules:
- apiGroups: ["config.openshift.io"]
  resources: ["infrastructures", "infrastructures/status"]
  verbs: ["get", "watch", "list"]
OpenTelemetry Collector using the Resource Detection processor
  config: |
    processor:
      resourcedetection:
        detectors: [openshift]
        override: true
    service:
      pipelines:
        traces:
          processors: [resourcedetection]
        metrics:
          processors: [resourcedetection]

Exporters

OTLP exporter

The OTLP gRPC exporter exports data using the OpenTelemetry protocol (OTLP).

OpenTelemetry Collector custom resource with an enabled OTLP exporter
  config: |
    exporters:
      otlp:
        endpoint: tempo-ingester:4317 (1)
        tls: (2)
          ca_file: ca.pem
          cert_file: cert.pem
          key_file: key.pem
        headers: (3)
          X-Scope-OrgID: "dev"
    service:
      pipelines:
        traces:
          exporters: [otlp]
        metrics:
          exporters: [otlp]
1 The OTLP gRPC endpoint. If the https:// scheme is used, then client transport security is enabled and overrides the insecure setting in the tls.
2 The client side TLS configuration. Defines paths to TLS certificates.
3 Headers are sent for every RPC performed during an established connection.
OTLP HTTP exporter

The OTLP HTTP exporter exports data using the OpenTelemetry protocol (OTLP).

OpenTelemetry Collector custom resource with an enabled OTLP exporter
  config: |
    exporters:
      otlphttp:
        endpoint: http://tempo-ingester:4318 (1)
        tls: (2)
          ca_file: ca.pem
          cert_file: cert.pem
          key_file: key.pem
        headers: (3)
          X-Scope-OrgID: "dev"

    service:
      pipelines:
        traces:
          exporters: [otlphttp]
        metrics:
          expoters: [otlphttp]
1 The OTLP HTTP endpoint. If the https:// scheme is used, then client transport security is enabled and overrides the insecure setting in the tls.
2 The client side TLS configuration. Defines paths to TLS certificates.
3 Headers are sent in every HTTP request.
Jaeger exporter

The Jaeger exporter exports data using the Jaeger proto format through gRPC.

OpenTelemetry Collector custom resource with enabled Jaeger exporter
  config: |
    exporters:
      jaeger:
        endpoint: jaeger-all-in-one:14250 (1)
        tls: (2)
          ca_file: ca.pem
          cert_file: cert.pem
          key_file: key.pem
    service:
      pipelines:
        traces:
          exporters: [jaeger]
1 The Jaeger gRPC endpoint.
2 The client side TLS configuration. Defines paths to TLS certificates.
Logging exporter

The Logging exporter prints data to the standard output.

OpenTelemetry Collector custom resource with an enabled Logging exporter
  config: |
    exporters:
      logging:
        verbosity: detailed (1)
    service:
      pipelines:
        traces:
          exporters: [logging]
        metrics:
          exporters: [logging]
1 Verbosity of the logging export: detailed or normal or basic. When set to detailed, pipeline data is verbosely logged. Defaults to normal.

Sending metrics to the monitoring stack

You can configure the monitoring stack to scrape OpenTelemetry Collector metrics endpoints and to remove duplicated labels that the monitoring stack has added during scraping.

Sample PodMonitor custom resource (CR) that configures the monitoring stack to scrape Collector metrics
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: otel-collector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: otel-collector
  podMetricsEndpoints:
  - port: metrics (1)
  - port: promexporter (2)
    relabelings:
    - action: labeldrop
      regex: pod
    - action: labeldrop
      regex: container
    - action: labeldrop
      regex: endpoint
    metricRelabelings:
    - action: labeldrop
      regex: instance
    - action: labeldrop
      regex: job
1 The name of the internal metrics port for the OpenTelemetry Collector. This port name is always metrics.
2 The name of the Prometheus exporter port for the OpenTelemetry Collector. This port name is defined in the .spec.ports section of the OpenTelemetryCollector CR.

Additional resources