Chapter 1. Configuring the instrumentation
The Red Hat build of OpenTelemetry Operator uses an Instrumentation custom resource that defines the configuration of the instrumentation.
1.1. Auto-instrumentation by using libraries
Auto-instrumentation in the Red Hat build of OpenTelemetry Operator can automatically instrument an application without manual code changes. Developers and administrators can monitor applications with minimal effort and changes to the existing codebase.
Auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Auto-instrumentation runs as follows:
- The Red Hat build of OpenTelemetry Operator injects an init-container, or a sidecar container for Go, to add the instrumentation libraries for the programming language of the instrumented application.
- The Red Hat build of OpenTelemetry Operator sets the required environment variables in the application’s runtime environment. These variables configure the auto-instrumentation libraries to collect traces, metrics, and logs and send them to the appropriate OpenTelemetry Collector or another telemetry backend.
- The injected libraries automatically instrument your application by connecting to known frameworks and libraries, such as web servers or database clients, to collect telemetry data. The source code of the instrumented application is not modified.
- Once the application is running with the injected instrumentation, the application automatically generates telemetry data, which is sent to a designated OpenTelemetry Collector or an external OTLP endpoint for further processing.
Auto-instrumentation enables you to start collecting telemetry data quickly without having to manually integrate the OpenTelemetry SDK into your application code. However, some applications might require specific configurations or custom manual instrumentation.
1.2. OpenTelemetry instrumentation configuration options
The Red Hat build of OpenTelemetry injects and configures the OpenTelemetry auto-instrumentation libraries into your workloads. Currently, the Red Hat build of OpenTelemetry supports injecting instrumentation libraries for Go, Java, Node.js, Python, .NET, and the Apache HTTP Server (httpd).
The Red Hat build of OpenTelemetry Operator only supports the injection mechanism of the instrumentation libraries but does not support instrumentation libraries or upstream images. Customers can build their own instrumentation images or use community images.
1.2.1. Instrumentation options
Instrumentation options are specified in an Instrumentation custom resource (CR).
The following Instrumentation CR example shows instrumentation options:
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: instrumentation
spec:
env:
- name: OTEL_EXPORTER_OTLP_TIMEOUT
value: "20"
exporter:
endpoint: http://production-collector.observability.svc.cluster.local:4317
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "1"
python:
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://production-collector.observability.svc.cluster.local:4318
dotnet:
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://production-collector.observability.svc.cluster.local:4318
go:
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://production-collector.observability.svc.cluster.local:4318where:
python- Python auto-instrumentation uses protocol buffers over HTTP (HTTP/proto or HTTP/protobuf) by default.
python.env-
Required if endpoint is set to
:4317. dotnet- .NET auto-instrumentation uses protocol buffers over HTTP (HTTP/proto or HTTP/protobuf) by default.
dotnet.env-
Required if endpoint is set to
:4317. go- Go auto-instrumentation uses protocol buffers over HTTP (HTTP/proto or HTTP/protobuf) by default.
go.envRequired if endpoint is set to
:4317.For more information about protocol buffers, see Overview (Protocol Buffers Documentation).
The Operator uses the following parameters to define the instrumentation:
| Parameter | Description | Values |
|---|---|---|
|
| Definition of common environment variables for all instrumentation types. | |
|
| Exporter configuration. | |
|
| Propagators defines inter-process context propagation configuration. |
|
|
| Resource attributes configuration. | |
|
| Sampling configuration. | |
|
| Configuration for the Apache HTTP Server instrumentation. | |
|
| Configuration for the .NET instrumentation. | |
|
| Configuration for the Go instrumentation. | |
|
| Configuration for the Java instrumentation. | |
|
| Configuration for the Node.js instrumentation. | |
|
| Configuration for the Python instrumentation. | Depending on the programming language, environment variables might not work for configuring telemetry. For the SDKs that do not support environment variable configuration, you must add a similar configuration directly in the code. For more information, see Environment Variable Specification (OpenTelemetry Documentation). |
The following protocols are the defaults for auto-instrumentation:
| Auto-instrumentation | Default protocol |
|---|---|
| Java 1.x |
|
| Java 2.x |
|
| Python |
|
| .NET |
|
| Go |
|
| Apache HTTP Server |
|
1.2.2. Configuration of the OpenTelemetry SDK variables
You can use the instrumentation.opentelemetry.io/inject-sdk annotation in the OpenTelemetryCollector custom resource. This annotation instructs the Red Hat build of OpenTelemetry Operator to inject some of the following OpenTelemetry SDK environment variables into your pod, depending on the Instrumentation CR:
-
OTEL_SERVICE_NAME -
OTEL_TRACES_SAMPLER -
OTEL_TRACES_SAMPLER_ARG -
OTEL_PROPAGATORS -
OTEL_RESOURCE_ATTRIBUTES -
OTEL_EXPORTER_OTLP_ENDPOINT -
OTEL_EXPORTER_OTLP_CERTIFICATE -
OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE -
OTEL_EXPORTER_OTLP_CLIENT_KEY
The instrumentation.opentelemetry.io/inject-sdk annotation accepts the following values:
| Value | Description |
|---|---|
|
|
Injects the |
|
|
Injects no |
|
|
Specifies the name of the |
|
|
Specifies the name of the |
1.2.3. Exporter configuration
Although the Instrumentation custom resource supports setting up one or more exporters per signal, auto-instrumentation configures only the OTLP Exporter. So you must configure the endpoint to point to the OTLP Receiver on the Collector.
The following exporter TLS CA configuration example uses a config map:
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
# ...
spec:
# ...
exporter:
endpoint: https://production-collector.observability.svc.cluster.local:4317
tls:
configMapName: ca-bundle
ca_file: service-ca.crt
# ...where:
endpoint- Specifies the OTLP endpoint using the HTTPS scheme and TLS.
configMapName- Specifies the name of the config map. The config map must already exist in the namespace of the pod injecting the auto-instrumentation.
ca_file- Points to the CA certificate in the config map or the absolute path to the certificate if the certificate is already present in the workload file system.
The following exporter mTLS configuration example uses a Secret:
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
# ...
spec:
# ...
exporter:
endpoint: https://production-collector.observability.svc.cluster.local:4317
tls:
secretName: serving-certs
ca_file: service-ca.crt
cert_file: tls.crt
key_file: tls.key
# ...where:
endpoint- Specifies the OTLP endpoint by using the HTTPS scheme and TLS.
secretName-
Specifies the name of the Secret for the
ca_file,cert_file, andkey_filevalues. The Secret must already exist in the namespace of the pod injecting the auto-instrumentation. ca_file- Points to the CA certificate in the Secret or the absolute path to the certificate if the certificate is already present in the workload file system.
cert_file- Points to the client certificate in the Secret or the absolute path to the certificate if the certificate is already present in the workload file system.
key_file- Points to the client key in the Secret or the absolute path to a key if the key is already present in the workload file system.
You can provide the CA certificate in a config map or Secret. If you provide it in both, the config map takes higher precedence than the Secret.
The following example shows the configuration for CA bundle injection by using a config map and Instrumentation CR:
apiVersion: v1
kind: ConfigMap
metadata:
name: otelcol-cabundle
namespace: tutorial-application
annotations:
service.beta.openshift.io/inject-cabundle: "true"
# ...
---
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
endpoint: https://simplest-collector.tracing-system.svc.cluster.local:4317
tls:
configMapName: otelcol-cabundle
ca: service-ca.crt
# ...1.2.4. Configuration of the Apache HTTP Server auto-instrumentation
The Apache HTTP Server auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
The .spec.apacheHttpd field uses the following parameters:
| Name | Description | Default |
|---|---|---|
|
| Attributes specific to the Apache HTTP Server. | |
|
| Location of the Apache HTTP Server configuration. |
|
|
| Environment variables specific to the Apache HTTP Server. | |
|
| Container image with the Apache SDK and auto-instrumentation. | |
|
| The compute resource requirements. | |
|
| Apache HTTP Server version. |
|
The following PodSpec annotation enables injection:
instrumentation.opentelemetry.io/inject-apache-httpd: "true"1.2.5. Configuration of the .NET auto-instrumentation
The .NET auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
By default, this feature injects unsupported, upstream instrumentation libraries.
| Name | Description |
|---|---|
|
| Environment variables specific to .NET. |
|
| Container image with the .NET SDK and auto-instrumentation. |
|
| The compute resource requirements. |
For the .NET auto-instrumentation, the required OTEL_EXPORTER_OTLP_ENDPOINT environment variable must be set if the endpoint of the exporters is set to 4317. The .NET autoinstrumentation uses http/proto by default, and the telemetry data must be set to the 4318 port.
The following PodSpec annotation enables injection:
instrumentation.opentelemetry.io/inject-dotnet: "true"1.2.6. Configuration of the Go auto-instrumentation
The Go auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
By default, this feature injects unsupported, upstream instrumentation libraries.
| Name | Description |
|---|---|
|
| Environment variables specific to Go. |
|
| Container image with the Go SDK and auto-instrumentation. |
|
| The compute resource requirements. |
The following PodSpec annotation enables injection:
instrumentation.opentelemetry.io/inject-go: "true"
instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/<path>/<to>/<container>/<executable>"where:
otel-go-auto-target-exe-
Sets the value for the required
OTEL_GO_AUTO_TARGET_EXEenvironment variable.
Go auto-instrumentation requires the following permissions in the OpenShift cluster:
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
name: otel-go-instrumentation-scc
allowHostDirVolumePlugin: true
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
allowedCapabilities:
- "SYS_PTRACE"
fsGroup:
type: RunAsAny
runAsUser:
type: RunAsAny
seLinuxContext:
type: RunAsAny
seccompProfiles:
- '*'
supplementalGroups:
type: RunAsAnyThe CLI command for applying the permissions for the Go auto-instrumentation in the OpenShift cluster is as follows:
$ oc adm policy add-scc-to-user otel-go-instrumentation-scc -z <service_account>1.2.7. Configuration of the Java auto-instrumentation
The Java auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
By default, this feature injects unsupported, upstream instrumentation libraries.
| Name | Description |
|---|---|
|
| Environment variables specific to Java. |
|
| Container image with the Java SDK and auto-instrumentation. |
|
| The compute resource requirements. |
The following PodSpec annotation enables injection:
instrumentation.opentelemetry.io/inject-java: "true"1.2.8. Configuration of the Node.js auto-instrumentation
The Node.js auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
By default, this feature injects unsupported, upstream instrumentation libraries.
| Name | Description |
|---|---|
|
| Environment variables specific to Node.js. |
|
| Container image with the Node.js SDK and auto-instrumentation. |
|
| The compute resource requirements. |
The following PodSpec annotations enable injection:
instrumentation.opentelemetry.io/inject-nodejs: "true"1.2.9. Configuration of the Python auto-instrumentation
The Python auto-instrumentation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
By default, this feature injects unsupported, upstream instrumentation libraries.
| Name | Description |
|---|---|
|
| Environment variables specific to Python. |
|
| Container image with the Python SDK and auto-instrumentation. |
|
| The compute resource requirements. |
For Python auto-instrumentation, the OTEL_EXPORTER_OTLP_ENDPOINT environment variable must be set if the endpoint of the exporters is set to 4317. Python auto-instrumentation uses http/proto by default, and the telemetry data must be set to the 4318 port.
The following PodSpec annotation enables injection:
instrumentation.opentelemetry.io/inject-python: "true"1.2.10. Multi-container pods
The instrumentation is injected to the first container that is available by default according to the pod specification. You can also specify the target container names for injection.
This pod annotation is required:
instrumentation.opentelemetry.io/container-names: "<container_1>,<container_2>"where:
container-names- Use this annotation when you want to inject a single instrumentation in multiple containers.
The Go auto-instrumentation does not support multi-container auto-instrumentation injection.
1.2.11. Multi-container pods with multiple instrumentations
Injecting instrumentation for an application language to one or more containers in a multi-container pod requires the following annotation:
instrumentation.opentelemetry.io/<application_language>-container-names: "<container_1>,<container_2>"where:
container-names-
You can inject instrumentation for only one language per container. For the list of supported
<application_language>values, see the following table.
The <application_language> parameter supports the following values:
| Language | Value for <application_language> |
|---|---|
| ApacheHTTPD |
|
| DotNet |
|
| Java |
|
| NGINX |
|
| NodeJS |
|
| Python |
|
| SDK |
|
1.2.12. Using the instrumentation CR with Service Mesh
When using the Instrumentation custom resource (CR) with Red Hat OpenShift Service Mesh, you must use the b3multi propagator.
1.3. Auto-instrumentation by using eBPF
Another auto-instrumentation tool is the zero-code OpenTelemetry eBPF Instrumentation (OBI).
OBI uses the extended Berkeley Packet Filter (eBPF) to automatically instrument supported Linux application executables at the kernel level and the operating system networking layer. OBI can capture trace spans, application metrics, runtime metrics, network metrics, and Rate Errors Duration (RED) metrics.
OBI is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
To implement OBI, use the OBI Receiver.
1.3.1. Collect traces and metrics without instrumenting your application
To collect traces and metrics without instrumenting your application, you can run the OBI Receiver, which uses Linux capabilities.
Prerequisites
-
The OBI Receiver requires nodes that use the
amd64orarm64CPU architecture. The OBI Receiver is not available on the IBM Z (s390x) or IBM Power (ppc64le) architectures.
Procedure
Apply the service account:
apiVersion: v1 kind: ServiceAccount metadata: name: obi-collector namespace: <namespace> # ...Apply the cluster role:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: obi rules: - apiGroups: ['apps'] resources: ['replicasets'] verbs: ['list', 'watch'] - apiGroups: [''] resources: ['pods', 'services', 'nodes'] verbs: ['list', 'watch']Apply the cluster role binding:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: obi subjects: - kind: ServiceAccount name: obi-collector namespace: <namespace> roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: obiApply the security context constraints:
apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: name: obi-unprivileged allowHostPorts: false allowHostDirVolumePlugin: true allowHostIPC: false allowHostNetwork: false allowHostPID: true allowPrivilegedContainer: false readOnlyRootFilesystem: true allowedCapabilities: - BPF - SYS_PTRACE - NET_RAW - CHECKPOINT_RESTORE - DAC_READ_SEARCH - PERFMON runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny volumes: - emptyDir - hostPath - configMap - secret - downwardAPI - projected users: - system:serviceaccount:<namespace>:obi-collector # ...where:
allowedCapabilities:-
The listed Linux capabilities load the eBPF programs, inspect processes and executables through the
/procfilesystem, attach socket filters to capture HTTP requests, and open the Executable and Linkable Format (ELF) files of the instrumented processes.
Apply the
OpenTelemetryCollectorcustom resource:apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: obi-collector namespace: <namespace> spec: mode: daemonset serviceAccount: obi-collector hostPID: true securityContext: runAsUser: 0 readOnlyRootFilesystem: true capabilities: add: - BPF - SYS_PTRACE - NET_RAW - CHECKPOINT_RESTORE - DAC_READ_SEARCH - PERFMON drop: - ALL volumes: - name: var-run-obi emptyDir: {} - name: cgroup hostPath: path: /sys/fs/cgroup type: Directory volumeMounts: - name: var-run-obi mountPath: /var/run/obi - name: cgroup mountPath: /sys/fs/cgroup readOnly: true config: receivers: obi: discovery: instrument: - k8s_namespace: <application_namespace> k8s_pod_labels: obi.instrument: "true" attributes: kubernetes: enable: "true" exporters: debug: verbosity: detailed service: pipelines: traces: receivers: [obi] exporters: [debug] # ...where:
hostPID: true- Enables the OBI Receiver to access the processes on the host.
discovery.instrument- Lists the processes the OBI Receiver attaches to. This example filters by namespace and pod labels, so that only pods matching all of the fields are instrumented.
k8s_namespace: <application_namespace>- Selects the workloads to instrument.
attributes.kubernetes.enable-
Setting this field to
"true"adds resource attributes such ask8s.namespace.name,k8s.pod.name, andservice.nameto the traces. Enables Kubernetes attribute enrichment. For enrichment, the CollectorServiceAccountrequires thelistandwatchpermissions on pods, services, and nodes in the core API group and on replica sets in theappsAPI group. These permissions are granted by theClusterRolein a previous step.
Add the matching label to the workload pod templates to opt them in:
apiVersion: apps/v1 kind: Deployment metadata: name: <application> namespace: <application_namespace> spec: template: metadata: labels: obi.instrument: "true" spec: containers: - name: <application> image: <application>:latest # ...
Verification
-
The OBI Receiver mounts the host
/sys/fs/cgroupdirectory as read-only so that it can monitor newly created sockets for outgoing requests.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}