1. Documentation
    2. history bug_report picture_as_pdf
×

Configuring and deploying the distributed tracing data collection

The Red Hat OpenShift distributed tracing data collection Operator uses a custom resource definition (CRD) file that defines the architecture and configuration settings to be used when creating and deploying the distributed tracing data collection resources. You can install the default configuration or modify the file.

OpenTelemetry Collector configuration options

The OpenTelemetry Collector consists of three components that access telemetry data:

Receivers

A receiver, which can be push or pull based, is how data gets into the Collector. Generally, a receiver accepts data in a specified format, translates it into the internal format, and passes it to processors and exporters defined in the applicable pipelines. By default, no receivers are configured. One or more receivers must be configured. Receivers may support one or more data sources.

Processors

Optional. Processors run through the data between it is received and exported. By default, no processors are enabled. Processors must be enabled for every data source. Not all processors support all data sources. Depending on the data source, multiple processors might be enabled. Note that the order of processors matters.

Exporters

An exporter, which can be push or pull based, is how you send data to one or more back ends or destinations. By default, no exporters are configured. One or more exporters must be configured. Exporters can support one or more data sources. Exporters might be used with their default settings, but many exporters require configuration to specify at least the destination and security settings.

You can define multiple instances of components in a custom resource YAML file. When configured, these components must be enabled through pipelines defined in the spec.config.service section of the YAML file. As a best practice, only enable the components that you need.

Example of the OpenTelemetry Collector custom resource file
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: cluster-collector
  namespace: tracing-system
spec:
  mode: deployment
  ports:
  - name: promexporter
    port: 8889
    protocol: TCP
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:
    exporters:
      jaeger:
        endpoint: jaeger-production-collector-headless.tracing-system.svc:14250
        tls:
          ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"
      prometheus:
        endpoint: 0.0.0.0:8889
        resource_to_telemetry_conversion:
          enabled: true # by default resource attributes are dropped
    service: (1)
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [jaeger]
        metrics:
          receivers: [otlp]
          processors: []
          exporters: [prometheus]
1 If a component is configured but not defined in the service section, the component is not enabled.
Table 1. Parameters used by the Operator to define the OpenTelemetry Collector
Parameter Description Values Default 
receivers:

A receiver is how data gets into the Collector. By default, no receivers are configured. There must be at least one enabled receiver for a configuration to be considered valid. Receivers are enabled by being added to a pipeline.

otlp, jaeger, zipkin

None

 
processors:

Processors run through the data between it is received and exported. By default, no processors are enabled.

None

 
exporters:

An exporter sends data to one or more back ends or destinations. By default, no exporters are configured. There must be at least one enabled exporter for a configuration to be considered valid. Exporters are enabled by being added to a pipeline. Exporters might be used with their default settings, but many require configuration to specify at least the destination and security settings.

otlp, otlphttp, jaeger, logging, prometheus

None

 
service:
  pipelines:

Components are enabled by adding them to a pipeline under services.pipeline.

 
service:
  pipelines:
    traces:
      receivers:

You enable receivers for tracing by adding them under service.pipelines.traces.

None

 
service:
  pipelines:
    traces:
      processors:

You enable processors for tracing by adding them under service.pipelines.traces.

None

 
service:
  pipelines:
    traces:
      exporters:

You enable exporters for tracing by adding them under service.pipelines.traces.

None

 
service:
  pipelines:
    metrics:
      receivers:

You enable receivers for metrics by adding them under service.pipelines.metrics.

None

 
service:
  pipelines:
    metrics:
      processors:

You enable processors for metircs by adding them under service.pipelines.metrics.

None

 
service:
  pipelines:
    metrics:
      exporters:

You enable exporters for metrics by adding them under service.pipelines.metrics.

None

OpenTelemetry Collector components

Receivers

OTLP Receiver

The OTLP receiver ingests data using the OpenTelemetry protocol (OTLP).

OpenTelemetry Collector custom resource with an enabled OTLP receiver
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317 (1)
            tls: (2)
              ca_file: ca.pem
              cert_file: cert.pem
              key_file: key.pem
              client_ca_file: client.pem (3)
              reload_interval: 1h (4)
          http:
            endpoint: 0.0.0.0:4318 (5)
            tls: (6)

    service:
      pipelines:
        traces:
          receivers: [otlp]
        metrics:
          receivers: [otlp]
1 The OTLP gRPC endpoint. If omitted, the default 0.0.0.0:4317 is used.
2 The server-side TLS configuration. Defines paths to TLS certificates. If omitted, TLS is disabled.
3 The path to the TLS certificate at which the server verifies a client certificate. This sets the value of ClientCAs and ClientAuth to RequireAndVerifyClientCert in the TLSConfig. For more information, see the Config of the Golang TLS package.
4 Specifies the time interval at which the certificate is reloaded. If the value is not set, the certificate is never reloaded. reload_interval accepts a string containing valid units of time such as ns, us (or µs), ms, s, m, h.
5 The OTLP HTTP endpoint. The default value is 0.0.0.0:4318.
6 The server-side TLS configuration. For more information, see grpc protocol configuration section.
Jaeger Receiver

The Jaeger receiver ingests data in Jaeger formats.

OpenTelemetry Collector custom resource with an enabled Jaeger receiver
  config: |
    receivers:
      jaeger:
        protocols:
          grpc:
            endpoint: 0.0.0.0:14250 (1)
          thrift_http:
            endpoint: 0.0.0.0:14268 (2)
          thrift_compact:
            endpoint: 0.0.0.0:6831 (3)
          thrift_binary:
            endpoint: 0.0.0.0:6832 (4)
          tls: (5)

    service:
      pipelines:
        traces:
          receivers: [jaeger]
1 The Jaeger gRPC endpoint. If omitted, the default 0.0.0.0:14250 is used.
2 The Jaeger Thrift HTTP endpoint. If omitted, the default 0.0.0.0:14268 is used.
3 The Jaeger Thrift Compact endpoint. If omitted, the default 0.0.0.0:6831 is used.
4 The Jaeger Thrift Binary endpoint. If omitted, the default 0.0.0.0:6832 is used.
5 The TLS server side configuration. See the OTLP receiver configuration section for more details.
Zipkin Receiver

The Zipkin receiver ingests data in the Zipkin v1 and v2 formats.

OpenTelemetry Collector custom resource with enabled Zipkin receiver
  config: |
    receivers:
      zipkin:
        endpoint: 0.0.0.0:9411 (1)
        tls: (2)

    service:
      pipelines:
        traces:
          receivers: [zipkin]
1 The Zipkin HTTP endpoint. If omitted, the default 0.0.0.0:9411 is used.
2 The TLS server side configuration. See the OTLP receiver configuration section for more details.

Processors

Batch processor

The batch processor batches the data to reduce the number of outgoing connections needed to transfer the telemetry information.

Example of the OpenTelemetry Collector custom resource when using the batch processor
  config: |
    processor:
      batch:
        timeout: 5s
        send_batch_max_size: 10000
    service:
      pipelines:
        traces:
          processors: [batch]
        metrics:
          processors: [batch]
Table 2. Parameters used by the batch processor
Parameter Description Default

timeout

Sends the batch after a specific time duration, irrespective of its size.

200ms

send_batch_size

Sends the batch of telemetry data after the specified number of spans or metrics.

8192

send_batch_max_size

The maximum allowable size of the batch. Must be equal or greater than send_batch_size.

0

metadata_keys

When activated, a batcher instance is created for each unique set of values found in the client.Metadata.

[]

metadata_cardinality_limit

When the metadata_keys are populated, this configuration restricts the number of distinct metadata key-value combinations processed throughout the duration of the process.

1000
Memory Limiter processor

The Memory Limiter processor periodically checks the Collector’s memory usage and pauses data processing when the soft memory limit is reached. The preceding component, which is typically a receiver, is expected to retry sending the same data and may apply a backpressure to the incoming data. When memory usage exceeds the hard limit, the Memory Limiter processor forces garbage collection to run.

  • Support level: General Availability

  • Supported signals: traces, metrics, logs

Example of the OpenTelemetry Collector custom resource when using the Memory Limiter processor
  config: |
    processor:
      memory_limiter:
        check_interval: 1s
        limit_mib: 4000
        spike_limit_mib: 800
    service:
      pipelines:
        traces:
          processors: [batch]
        metrics:
          processors: [batch]
Table 3. Parameters used by the Memory Limiter processor
Parameter Description Default

check_interval

Time between memory usage measurements. The optimal value is 1s. For spiky traffic patterns, you can decrease the check_interval or increase the spike_limit_mib.

0s

limit_mib

The hard limit, which is the maximum amount of memory in MiB allocated on the heap. Typically, the total memory usage of the OpenTelemetry Collector is about 50 MiB greater than this value.

0

spike_limit_mib

Spike limit, which is the maximum expected spike of memory usage in MiB. The optimal value is approximately 20% of limit_mib. To calculate the soft limit, subtract the spike_limit_mib from the limit_mib.

20% of limit_mib

limit_percentage

Same as the limit_mib but expressed as a percentage of the total available memory. The limit_mib setting takes precedence over this setting.

0

spike_limit_percentage

Same as the spike_limit_mib but expressed as a percentage of the total available memory. Intended to be used with the limit_percentage setting.

0
Resource Detection processor

The Resource Detection processor is designed to identify host resource details in alignment with OpenTelemetry’s resource semantic standards. Using this detected information, it can add or replace the resource values in telemetry data.

OpenShift Container Platform permissions required for the Resource Detection processor
kind: ClusterRole
metadata:
  name: otel-collector
rules:
- apiGroups: ["config.openshift.io"]
  resources: ["infrastructures", "infrastructures/status"]
  verbs: ["get", "watch", "list"]
OpenTelemetry Collector using the Resource Detection processor
  config: |
    processor:
      resourcedetection:
        detectors: [openshift]
        override: true
    service:
      pipelines:
        traces:
          processors: [resourcedetection]
        metrics:
          processors: [resourcedetection]

Exporters

OTLP exporter

The OTLP gRPC exporter exports data using the OpenTelemetry protocol (OTLP).

OpenTelemetry Collector custom resource with an enabled OTLP exporter
  config: |
    exporters:
      otlp:
        endpoint: tempo-ingester:4317 (1)
        tls: (2)
          ca_file: ca.pem
          cert_file: cert.pem
          key_file: key.pem
          insecure: false (3)
          insecure_skip_verify: false (4)
          reload_interval: 1h (5)
          server_name_override: <name> (6)
        headers: (7)
          X-Scope-OrgID: "dev"
    service:
      pipelines:
        traces:
          exporters: [otlp]
        metrics:
          exporters: [otlp]
1 The OTLP gRPC endpoint. If the https:// scheme is used, then client transport security is enabled and overrides the insecure setting in the tls.
2 The client side TLS configuration. Defines paths to TLS certificates.
3 Disables client transport security when set to true. The default value is false by default.
4 Skips verifying the certificate when set to true. The default value is false.
5 Specifies the time interval at which the certificate is reloaded. If the value is not set, the certificate is never reloaded. reload_interval accepts a string containing valid units of time such as ns, us (or µs), ms, s, m, h.
6 Overrides the virtual host name of authority such as the authority header field in requests. You can use this for testing.
7 Headers are sent for every request performed during an established connection.
OTLP HTTP exporter

The OTLP HTTP exporter exports data using the OpenTelemetry protocol (OTLP).

OpenTelemetry Collector custom resource with an enabled OTLP exporter
  config: |
    exporters:
      otlphttp:
        endpoint: http://tempo-ingester:4318 (1)
        tls: (2)
        headers: (3)
          X-Scope-OrgID: "dev"

    service:
      pipelines:
        traces:
          exporters: [otlphttp]
        metrics:
          expoters: [otlphttp]
1 The OTLP HTTP endpoint. If the https:// scheme is used, then client transport security is enabled and overrides the insecure setting in the tls.
2 The client side TLS configuration. Defines paths to TLS certificates.
3 Headers are sent in every HTTP request.
Jaeger exporter

The Jaeger exporter exports data using the Jaeger proto format through gRPC.

OpenTelemetry Collector custom resource with enabled Jaeger exporter
  config: |
    exporters:
      jaeger:
        endpoint: jaeger-all-in-one:14250 (1)
        tls: (2)
    service:
      pipelines:
        traces:
          exporters: [jaeger]
1 The Jaeger gRPC endpoint.
2 The client side TLS configuration. Defines paths to TLS certificates.
Logging exporter

The Logging exporter prints data to the standard output.

OpenTelemetry Collector custom resource with an enabled Logging exporter
  config: |
    exporters:
      logging:
        verbosity: detailed (1)
    service:
      pipelines:
        traces:
          exporters: [logging]
        metrics:
          exporters: [logging]
1 Verbosity of the logging export: detailed or normal or basic. When set to detailed, pipeline data is verbosely logged. Defaults to normal.
Prometheus exporter

The Prometheus exporter exports data using the Prometheus or OpenMetrics formats.

OpenTelemetry Collector custom resource with an enabled Prometheus exporter
  ports:
  - name: promexporter (1)
    port: 8889
    protocol: TCP
  config: |
    exporters:
      prometheus:
        endpoint: 0.0.0.0:8889 (2)
        tls: (3)
          ca_file: ca.pem
          cert_file: cert.pem
          key_file: key.pem
        namespace: prefix (4)
        const_labels: (5)
          label1: value1
        enable_open_metrics: true (6)
        resource_to_telemetry_conversion: (7)
          enabled: true
        metric_expiration: 180m (8)
    service:
      pipelines:
        metrics:
          exporters: [prometheus]
1 Exposes the Prometheus port from the collector pod and service. You can enable scraping of metrics by Prometheus by using the port name in ServiceMonitor or PodMonitor custom resource.
2 The network endpoint where the metrics are exposed.
3 The server-side TLS configuration. Defines paths to TLS certificates.
4 If set, exports metrics under the provided value. No default.
5 Key-value pair labels that are applied for every exported metric. No default.
6 If true, metrics are exported using the OpenMetrics format. Exemplars are only exported in the OpenMetrics format and only for histogram and monotonic sum metrics such as counter. Disabled by default.
7 If enabled is true, all the resource attributes are converted to metric labels by default. Disabled by default.
8 Defines how long metrics are exposed without updates. The default is 5m.

Sending metrics to the monitoring stack

You can configure the monitoring stack to scrape OpenTelemetry Collector metrics endpoints and to remove duplicated labels that the monitoring stack has added during scraping.

Sample PodMonitor custom resource (CR) that configures the monitoring stack to scrape Collector metrics
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: otel-collector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: otel-collector
  podMetricsEndpoints:
  - port: metrics (1)
  - port: promexporter (2)
    relabelings:
    - action: labeldrop
      regex: pod
    - action: labeldrop
      regex: container
    - action: labeldrop
      regex: endpoint
    metricRelabelings:
    - action: labeldrop
      regex: instance
    - action: labeldrop
      regex: job
1 The name of the internal metrics port for the OpenTelemetry Collector. This port name is always metrics.
2 The name of the Prometheus exporter port for the OpenTelemetry Collector. This port name is defined in the .spec.ports section of the OpenTelemetryCollector CR.

Additional resources