Chapter 1. Centralize your observability stack with OpenTelemetry
Learn about OpenTelemetry integration for centralized observability and telemetry data.
OpenTelemetry (OTel) is an open-source observability framework designed to provide consistent and reliable ways to collect telemetry data across distributed systems. It is a project under the Cloud Native Computing Foundation (CNCF) as an incubating project. Its goal is to provide a unified standard and toolset for instrumenting, generating, and exporting telemetry data from modern applications, making it easier to monitor, debug, and understand complex services.
The goal of Red Hat build of Keycloak is to support OpenTelemetry Logs, Metrics, and Traces to create a centralized observability stack without any additional deployment overhead.
1.1. General options
Red Hat build of Keycloak provides support for global settings of telemetry data export. It means that you can configure connection parameters to a single OpenTelemetry collector that will be used for all supported components.
In order to have OpenTelemetry enabled, at least one component needs to be turned on, for example Traces.
The OpenTelemetry feature is enabled by default. Disable the feature in order to disable OpenTelemetry components.
1.1.1. Endpoint
You can configure the OpenTelemetry collector’s global endpoint that is shared across all components by using the telemetry-endpoint option. The default value is http://localhost:4317.
You can change the endpoint via CLI as follows:
bin/kc.[sh|bat] start --telemetry-endpoint=http://my-otel-endpoint:43171.1.2. Service name
You can configure the global OpenTelemetry service name used for identifying the exporter of the telemetry data. The default service name is keycloak, specified via the telemetry-service-name property, which takes precedence over service.name defined in the telemetry-resource-attributes property.
You can change the service name via CLI as follows:
bin/kc.[sh|bat] start --telemetry-service-name=my-keycloak-iam1.1.3. Protocol
You can configure the global OpenTelemetry transport protocol used as a communication channel between Red Hat build of Keycloak and the OpenTelemetry collector. The default value is grpc.
Red Hat build of Keycloak supports these communication protocols:
-
grpc(default) -
http/protobuf
You can change the protocol via CLI as follows:
bin/kc.[sh|bat] start --telemetry-protocol=http/protobuf1.1.4. Request headers
You can configure custom request headers that will be sent when exporting telemetry data to the OpenTelemetry collector. This is useful for providing authentication tokens or other custom headers required by your telemetry backend.
Use the telemetry-header-<header> option to set headers for all telemetry components (logs, metrics, and traces).
Replace <header> with your custom header name in the option name.
bin/kc.[sh|bat] start --telemetry-header-Authorization='Bearer my-token' --telemetry-header-X-Custom-Header=custom-valueComponent-specific headers take precedence over general headers for their respective components.
1.2. Configuration via Keycloak CR (Operator)
The Keycloak CR has first-class citizen configuration options for telemetry. These may be configured under the spec.telemtry stanza as follows:
apiVersion: k8s.keycloak.org/v2beta1
kind: Keycloak
metadata:
name: example-kc
spec:
telemetry:
endpoint: http://my-telemetry:4317 # default 'http://localhost:4317'
serviceName: my-best-keycloak-telemetry # default 'keycloak'
protocol: http/protobuf # default 'grpc'
resourceAttributes:
service.namespace: keycloak-namespace-telemetry
These fields should reflect a 1:1 association with telemetry-* server options.
1.2.1. Enable all OpenTelemetry components
To enable OpenTelemetry metrics, logs, and traces, you can configure the Keycloak CR as follows:
OpenTelemetry Logs and Metrics are preview/experimental features.
apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
name: example-kc
spec:
features:
enabled:
- opentelemetry-logs
- opentelemetry-metrics
telemetry:
endpoint: http://my-telemetry:4317
serviceName: my-best-keycloak-telemetry
protocol: grpc
tracing:
enabled: true
additionalOptions:
- name: telemetry-logs-enabled
value: "true"
- name: telemetry-metrics-enabled
value: "true"
- name: metrics-enabled
value: "true"1.3. Traces
See the Root cause analysis with tracing guide.
You can override the global OpenTelemetry settings via the Tracing options if you want to export traces to a different OpenTelemetry collector.
1.4. Logs
OpenTelemetry Logs support is Technology Preview and is not fully supported. This feature is disabled by default.
To enable start the server with --features=preview or --features=opentelemetry-logs
The OpenTelemetry feature (opentelemetry) needs to be turned on (by default).
It is possible to export Red Hat build of Keycloak’s logs to the OpenTelemetry collector and be managed by various logging backends that support OpenTelemetry.
1.4.1. Enable Logs
You can enable OpenTelemetry Logs via CLI as follows:
bin/kc.[sh|bat] start --features=opentelemetry-logs --telemetry-logs-enabled=trueFor more information on how to set up logging, see the Configuring Logging guide.
1.4.2. Log level
By default, all log levels are exported by Red Hat build of Keycloak. You can optionally configure the most verbose log level to be exported via the telemetry-logs-level option.
The telemetry-logs-level option filters which logs are exported to OpenTelemetry, while log-level and log-level-<category> control which logs Red Hat build of Keycloak generates.
The telemetry filter is applied to already-generated logs, so setting telemetry-logs-level=DEBUG will not export DEBUG logs if log-level=WARN.
For example, if you want only to export WARN and ERROR logs, you can change the log level via the CLI as follows:
bin/kc.[sh|bat] start --telemetry-logs-level=WARN1.4.3. Request headers
As stated for the general configuration of headers, you can configure custom request headers that will be used when exporting logs.
bin/kc.[sh|bat] start --telemetry-logs-header-Authorization='Bearer logs-token'1.5. Metrics
The OpenTelemetry Metrics feature is currently experimental, and it is not recommended for use in production.
In order for the Metrics feature to function, the OpenTelemetry feature opentelemetry cannot be disabled.
For integrating OpenTelemetry Metrics, Red Hat build of Keycloak uses the Micrometer-to-OpenTelemetry bridge, which provides the functionality to export metrics created by Micrometer to the OpenTelemetry Collector. This means that all metrics created via Micrometer or OpenTelemetry metrics will be exported to the OpenTelemetry.
There are some API, and Semantic convention differences for Micrometer vs OTel Metrics, so you should check if all required metrics are exported.
1.5.1. Enable Metrics
You can enable the OpenTelemetry Metrics via CLI as follows:
bin/kc.[sh|bat] start --features=opentelemetry-metrics --telemetry-metrics-enabled=true --metrics-enabled=true
Metrics (metrics-enabled) needs to be enabled as well
For more information on how to set up metrics, see the configuration options below or visit the Gaining insights with metrics guide.
1.5.2. Request headers
As stated for the general configuration of headers, you can configure custom request headers that will be used when exporting metrics.
bin/kc.[sh|bat] start --telemetry-metrics-header-Authorization='Bearer metrics-token'1.6. Relevant options
| Type or Values | Default | |
|---|---|---|
|
Available only when any of available OpenTelemetry components (Logs, Metrics, Traces) is turned on | String |
|
|
Available only when any of available OpenTelemetry components (Logs, Metrics, Traces) is turned on | String | |
|
Available only when any of available OpenTelemetry components (Logs, Metrics, Traces) is turned on |
|
|
|
Available only when any of available OpenTelemetry components (Logs, Metrics, Traces) is turned on | List | |
|
Available only when any of available OpenTelemetry components (Logs, Metrics, Traces) is turned on | String |
|
1.6.1. Traces
| Type or Values | Default | |
|---|---|---|
| 🛠
Available only when 'opentelemetry' feature is enabled |
|
|
|
Available only when Tracing is enabled | String |
|
|
Available only when Tracing is enabled |
|
|
|
Available only when Tracing is enabled
DEPRECATED. Resource attributes are not directly related to Tracing and you should use the Telemetry option which takes precedence. Use: | List | |
|
Available only when Tracing is enabled
DEPRECATED. Service name is not directly related to Tracing and you should use the Telemetry option which takes precedence. Use: | String |
|
1.6.2. Logs
| Type or Values | Default | |
|---|---|---|
| 🛠
Available only when feature 'opentelemetry-logs:v1' is enabled |
|
|
|
Available only when Telemetry Logs functionality ('telemetry-logs-enabled') is enabled | String | |
|
Available only when Telemetry Logs functionality ('telemetry-logs-enabled') is enabled | String | |
|
Available only when Telemetry Logs functionality ('telemetry-logs-enabled') is enabled |
|
|
|
Available only when Telemetry Logs functionality ('telemetry-logs-enabled') is enabled |
|
1.6.3. Metrics
| Type or Values | Default | |
|---|---|---|
| 🛠
|
|
|
| 🛠
Available only when metrics and feature 'opentelemetry-metrics:v1' are enabled |
|
|
|
Available only when metrics ('metrics-enabled') and Telemetry Metrics functionality ('telemetry-metrics-enabled') are enabled | String | |
|
Available only when metrics ('metrics-enabled') and Telemetry Metrics functionality ('telemetry-metrics-enabled') are enabled | String | |
|
Available only when metrics ('metrics-enabled') and Telemetry Metrics functionality ('telemetry-metrics-enabled') are enabled | String |
|
|
Available only when metrics ('metrics-enabled') and Telemetry Metrics functionality ('telemetry-metrics-enabled') are enabled |
|
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}