SupportConsoleDevelopersStart a trialContact

Chapter 5. Forward to Kafka

You can forward logs to a Kafka broker.

5.1. Forward logs to a Kafka broker
Copy link

To configure log forwarding to an external Kafka instance, you must create a ClusterLogForwarder custom resource (CR) with an output to that instance, and a pipeline that uses the output. You can include a specific Kafka topic in the output or use the default. The Kafka output can use a TCP (insecure) or TLS (secure TCP) connection.

Procedure

  1. Create or edit a YAML file that defines the ClusterLogForwarder CR object: 

    apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: <log_forwarder_name>
  namespace: <log_forwarder_namespace>
spec:
  serviceAccount:
    name: <service_account_name>
  outputs:
   - name: kafka-output
     type: kafka
     kafka:
       authentication:
         sasl:
           username:
             key: <key>
             secretName: kafka-secret
           password:
            key: <key>
            secretName:  kafka-secret
           mechanism: <sasl_mechanism>
       url: tls://kafka.example.example.com:9093/app-topic
       brokers:
         - tls://kafka-broker1.example.com:9093
         - tls://kafka-broker2.example.com:9093
       topic:
  pipelines:
    - name: app-topic
      inputRefs:
        - application
      outputRefs:
        - kafka-output
    serviceAccount.name
    Specify the name of your service account.
    outputs.name
    Specify a name for the output.
    type
    Specify the kafka type.
    secretName
    If you use a tls prefix in the URL, you must specify the name of the secret required by the endpoint for TLS communication.
    mechanism
    Specify the Simple Authentication and Security Layer (SASL) mechanism to use. For example, SCRAM-SHA-256,SCRAM-SHA-512, or PLAIN. The default value is PLAIN.
    url
    Optional: Specify the URL and port of the Kafka broker as a valid absolute URL, optionally with a specific topic. You can use the tcp (insecure) or tls (secure TCP) protocol. If you enable the cluster-wide proxy by using the CIDR annotation, the output must be a server name or FQDN, and not an IP address. You must specify either the URL or Kafka brokers.
    brokers
    Optional: Specify a list of broker endpoints of a Kafka cluster.
    topic
    Specify the target topic. By default, the value for the field is topic. The topic name can be a combination of static and dynamic values consisting of field paths separated by "||". Encase dynamic values in single curly brackets "{}". Follow dynamic values with a static fallback value. The topic specified here overrides the topic defined in the url field.

  2. Apply the ClusterLogForwarder CR by running the following command: 

    $ oc apply -f <filename>.yaml

5.2. Kafka message structure and field mappings
Copy link

When you forward logs to Apache Kafka, the log collector transforms OpenShift log records into Kafka messages with three components: key, value, and headers. Understanding this structure helps you configure Kafka consumers and optimize partition distribution.

Kafka messages have three components that you control through the ClusterLogForwarder output configuration:

key
Optional field extracted from the log record. Kafka uses the hash of the key to determine the partition. If you do not configure a key, Kafka uses round-robin distribution across partitions. Configure this using the key_field parameter.
value
The log record payload, typically JSON-encoded. By default, the collector sends the entire log record as JSON. You can filter fields by using encoding.only_fields or encoding.except_fields.
headers
Optional key-value pairs for metadata. Configure this by using the headers_key parameter to specify a log record field containing the header data.

The following table shows how ClusterLogForwarder configuration controls the Kafka message structure:

Configuration fieldKafka componentDescription

key_field

Message key

Specifies which log record field to use as the Kafka message key

encoding.codec

Message value

Controls the format of the message value (default: json)

encoding.only_fields or encoding.except_fields

Message value

Filters which fields appear in the message value

headers_key

Message headers

Specifies which log record field contains key-value pairs for Kafka headers 

{
  "kubernetes": {
    "namespace_name": "my-app",
    "pod_name": "web-1",
    "container_name": "nginx"
  },
  "message": "Error connecting to database",
  "level": "error",
  "@timestamp": "2024-01-15T10:30:00Z",
  "kafka_headers": {
    "source": "openshift",
    "cluster": "prod-us-west"
  }
}

With key_field: "kubernetes.namespace_name" and headers_key: "kafka_headers", the log collector produces the following Kafka message: 

Topic: openshift-logs
Partition: 3 (hash of "my-app")

Key: "my-app"

Value:
{
  "kubernetes": {
    "namespace_name": "my-app",
    "pod_name": "web-1",
    "container_name": "nginx"
  },
  "message": "Error connecting to database",
  "level": "error",
  "@timestamp": "2024-01-15T10:30:00Z"
}

Headers:
  source: "openshift"
  cluster: "prod-us-west"

The relationship between log events and Kafka messages is one-to-one. Each log record produces exactly one Kafka message. The collector’s batching configuration affects only the efficiency of send operations to Kafka brokers, not the number of messages.

The topic field supports template syntax, allowing you to route logs to different topics based on log record fields. For example, topic: "{{ kubernetes.namespace_name }}" creates a separate topic for each namespace.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top