Chapter 12. Forward from pods and projects
You can configure log forwarding to send logs from specific pods or projects, allowing you to route logs based on application labels or namespace ownership.
12.1. Forward application logs from specific pods Copy linkLink copied to clipboard!
As a cluster administrator, you can use Kubernetes pod labels to gather log data from specific pods and forward it to a log collector.
Suppose that you have an application composed of pods running alongside other pods in various namespaces. If those pods have labels that identify the application, you can gather and output their log data to a specific log collector.
To specify the pod labels, you use one or more matchLabels key-value pairs. If you specify many key-value pairs, the collector selects only pods that match all of them.
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarderCR object. In the file, specify the pod labels using simple equality-based selectors underinputs[].name.application.selector.matchLabels, as shown in the following example.apiVersion: observability.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> namespace: <log_forwarder_namespace> spec: serviceAccount: name: <service_account_name> outputs: - <output_name> # ... inputs: - name: exampleAppLogData type: application application: includes: - namespace: app1 - namespace: app2 selector: matchLabels: environment: production app: nginx pipelines: - inputRefs: - exampleAppLogData outputRefs: # ...serviceAccount.name- Specify the service account name.
inputs.name- Specify a name for the input.
type-
Specify the type as
applicationto collect logs from applications. includes- Specify the set of namespaces to include when collecting logs.
matchLabels- Specify the key-value pairs of pod labels whose log data you want to gather. You must specify both a key and value, not just a key. The collector selects only pods that match all the key-value pairs.
Optional: You can send log data from additional applications that have different pod labels to the same pipeline.
-
For each unique combination of pod labels, create an additional
inputs[].namesection similar to the one shown. -
Update the
selectorsto match the pod labels of this application. Add the new
inputs[].namevalue toinputRefs. For example:- inputRefs: [ myAppLogData, myOtherAppLogData ]
-
For each unique combination of pod labels, create an additional
Create the CR object:
$ oc create -f <file_name>.yaml
12.1.1. Forward logs by using the syslog protocol Copy linkLink copied to clipboard!
You can use the syslog RFC3164 or RFC5424 protocol to send a copy of your logs to an external log aggregator. You are responsible for configuring the external log aggregator, such as a syslog server, to receive the logs from OpenShift Container Platform.
To configure log forwarding by using the syslog protocol, you must create a ClusterLogForwarder custom resource (CR) with one or more outputs to the syslog servers, and pipelines that use those outputs. The syslog output can use a UDP, TCP, or TLS connection.
Prerequisites
- You must configure a logging server to receive the logging data by using the specified protocol or format.
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarderCR object:apiVersion: observability.openshift.io/v1 kind: ClusterLogForwarder metadata: name: collector spec: managementState: Managed outputs: - name: rsyslog-east syslog: appName: <app_name> enrichment: KubernetesMinimal facility: <facility_value> msgId: <message_ID> payloadKey: <record_field> procId: <process_ID> rfc: <RFC3164_or_RFC5424> severity: informational tuning: deliveryMode: <AtLeastOnce_or_AtMostOnce> url: <url> tls: ca: key: ca-bundle.crt secretName: syslog-secret type: syslog pipelines: - inputRefs: - application name: syslog-east outputRefs: - rsyslog-east serviceAccount: name: logcollectoroutputs.name- Specify a name for the output.
appName-
Optional: Specify the value for the
APP-NAMEpart of the syslog message header. The value must conform with The Syslog Protocol. The value can be a combination of static and dynamic values consisting of field paths followed by||, and then followed by another field path or a static value. The collector truncates the final values to a maximum length of 48 characters. You must encase a dynamic value in curly brackets and follow the value with a static fallback value separated with||. Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes. Example value: <value1>-{.<value2>||"none"}. enrichment-
Optional: Specify the enrichment level for Kubernetes metadata fields added to log records. The value can be
KubernetesMinimalto add minimal Kubernetes fields such as namespace, pod name, and container name. This parameter replaces theaddLogSourceparameter that was removed in Logging 6.0. facility-
Optional: Specify the value for
Facilitypart of the syslog-msg header. msgId-
Optional: Specify the value for
MSGIDpart of the syslog-msg header. The value can be a combination of static and dynamic values consisting of field paths followed by||, and then followed by another field path or a static value. The collector truncates the final values to a maximum length of 32 characters. You must encase a dynamic value in curly brackets and follow the value with a static fallback value separated with||. Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes. Example value: <value1>-{.<value2>||"none"}. payloadKey-
Optional: Specify the record field to use as the payload. The
payloadKeyvalue must be a single field path encased in single curly brackets{}. Example: {.<value>}. procId-
Optional: Specify the value for the
PROCIDpart of the syslog message header. The value must conform with The Syslog Protocol. The value can be a combination of static and dynamic values consisting of field paths followed by||, and then followed by another field path or a static value. The collector truncates the final values to a maximum length of 48 characters. You must encase a dynamic value in curly brackets and follow the value with a static fallback value separated with||. Static values can only contain alphanumeric characters along with dashes, underscores, dots and forward slashes. Example value: <value1>-{.<value2>||"none"}. rfc-
Optional: Set the RFC that the generated messages conform to. The value can be
RFC3164orRFC5424. severity- Optional: Set the severity level for the message. For more information, see The Syslog Protocol.
deliveryMode-
Optional: Set the delivery mode for log forwarding. The value can be either
AtLeastOnce, orAtMostOnce. url-
Specify the absolute URL with a scheme. Valid schemes are:
tcp,tls, andudp. For example:tls://syslog-receiver.example.com:6514. tls- Specify the settings for controlling options of the transport layer security (TLS) client connections.
inputRefs-
Specify which log types to forward by using the pipeline:
application,infrastructure, oraudit. pipelines.name- Specify a name for the pipeline.
serviceAccount- The name of your service account.
Create the CR object:
$ oc create -f <filename>.yaml
12.1.2. Default field mappings for RFC5424 Copy linkLink copied to clipboard!
When using the RFC5424 protocol, the collector automatically populates the APP-NAME, PROCID, and MSGID fields with default values based on the log source type. You can override these defaults by specifying appName, procId, or msgId in the syslog output configuration.
The following table shows the default field mappings:
| Log source | APP-NAME | PROCID | MSGID |
|---|---|---|---|
| Infrastructure (journal logs) |
|
|
|
| Infrastructure (container logs) |
|
|
|
| Application (container logs) |
|
|
|
| Audit logs |
|
|
|
Field values can use template syntax for dynamic per-event values. The maximum length for APP-NAME and PROCID is 48 characters, and for MSGID is 32 characters. The collector truncates values that exceed these limits.
12.1.3. Syslog output format and field mappings Copy linkLink copied to clipboard!
When you forward logs to a syslog server, the log collector transforms OpenShift log records into RFC 5424 or RFC 3164 syslog format. Understanding this transformation helps you configure syslog server parsing rules and log routing.
The log collector sends logs by using the syslog protocol over TCP or UDP. You can configure the RFC format, severity mapping, and facility in the ClusterLogForwarder output configuration.
12.1.3.1. RFC 5424 format (recommended) Copy linkLink copied to clipboard!
RFC 5424 is the modern syslog format that supports structured data. The collector uses this format by default.
RFC 5424 message structure:
<PRIORITY>VERSION TIMESTAMP HOSTNAME APP-NAME PROCID MSGID [STRUCTURED-DATA] MESSAGE
The following table shows how OpenShift log record fields map to RFC 5424 components:
| RFC 5424 field | OpenShift field | Notes |
|---|---|---|
|
|
Calculated: | Default facility: 1 (user-level). Severity mapped from log level. |
|
|
Static: | RFC 5424 version |
|
|
| ISO 8601 format with time zone |
|
|
| Kubernetes node name |
|
| Log type-dependent |
|
|
| Log type-dependent |
|
|
| Not used |
Set to |
|
| Kubernetes metadata | Custom structured data elements with namespace, pod, container information |
|
|
| UTF-8 encoded log message |
The following OpenShift log record:
{
"kubernetes": {
"namespace_name": "my-app",
"pod_name": "web-1",
"container_name": "nginx"
},
"message": "Error connecting to database",
"level": "error",
"@timestamp": "2024-01-15T10:30:00.000Z",
"hostname": "ip-10-0-1-42.ec2.internal"
}
The log collector transforms this into RFC 5424 syslog:
<11>1 2024-01-15T10:30:00.000Z ip-10-0-1-42.ec2.internal my-app web-1 - [kubernetes namespace_name="my-app" pod_name="web-1" container_name="nginx"] Error connecting to database
Priority calculation: (1 * 8) + 3 = 11 (facility 1 = user, severity 3 = error)
12.1.3.2. RFC 3164 format (legacy) Copy linkLink copied to clipboard!
RFC 3164 is the older syslog format without structured data support. Use this format for compatibility with legacy syslog servers.
RFC 3164 message structure:
<PRIORITY>TIMESTAMP HOSTNAME TAG: MESSAGE
The following table shows how OpenShift log record fields map to RFC 3164 components:
| RFC 3164 field | OpenShift field | Notes |
|---|---|---|
|
|
Calculated: | Default facility: 1 (user-level). Severity mapped from log level. |
|
|
|
MMM DD HH:MM:SS format (for example, |
|
|
| Kubernetes node name |
|
| Log type-dependent | Namespace/pod for application logs, process name for infrastructure logs |
|
|
| UTF-8 encoded log message |
The same OpenShift log record in RFC 3164 format:
<11>Jan 15 10:30:00 ip-10-0-1-42.ec2.internal my-app/web-1: Error connecting to database
12.1.3.3. Severity mapping Copy linkLink copied to clipboard!
The log collector maps OpenShift log levels to syslog severity values:
| OpenShift level | Syslog severity | Value |
|---|---|---|
|
| Debug | 7 |
|
| Informational | 6 |
|
| Warning | 4 |
|
| Error | 3 |
|
| Critical | 2 |
|
| Alert | 1 |
|
| Emergency | 0 |
If the log record does not have a level field, the collector defaults to severity 6 (Informational).
12.1.3.4. Facility codes Copy linkLink copied to clipboard!
The syslog facility indicates the source of the log message. You can configure the facility in the ClusterLogForwarder output configuration.
Common facility values:
| Value | Facility | Description |
|---|---|---|
| 0 | kern | kernel messages |
| 1 | user | User-level messages (default) |
| 3 | daemon | System daemons |
| 4 | auth | Security/authentication messages |
| 16-23 | local0-local7 | Locally-defined facilities |
The default facility is 1 (user-level). For infrastructure logs, you might configure facility 3 (daemon) or a local facility.
The following infrastructure log from systemd journal:
{
"message": "Started container",
"SYSLOG_IDENTIFIER": "kubelet",
"hostname": "ip-10-0-1-42.ec2.internal",
"@timestamp": "2024-01-15T10:30:00.000Z"
}
The log collector transforms this into RFC 5424 syslog:
<14>1 2024-01-15T10:30:00.000Z ip-10-0-1-42.ec2.internal kubelet - - - Started container
Priority calculation: (1 * 8) + 6 = 14 (facility 1 = user, severity 6 = info)
12.1.3.5. Transport protocols Copy linkLink copied to clipboard!
The log collector supports the following transport protocols for syslog:
- TCP
- Reliable delivery with connection-oriented transport. Recommended for production environments.
- UDP
- Connectionless transport that reduces network traffic but provides no delivery guarantees. Use for high-volume, non-critical logs.
- TLS over TCP
- Encrypted transport with certificate-based authentication. Required for secure log forwarding across untrusted networks.
Configure the transport protocol in the ClusterLogForwarder output specification.
12.1.4. Add log source information to the message output Copy linkLink copied to clipboard!
You can add namespace_name, pod_name, and container_name elements to the message field of the record by adding the enrichment field to your ClusterLogForwarder custom resource (CR).
# ...
spec:
outputs:
- name: syslogout
syslog:
enrichment: KubernetesMinimal
facility: user
payloadKey: message
rfc: RFC3164
severity: debug
type: syslog
url: tls://syslog-receiver.example.com:6514
pipelines:
- inputRefs:
- application
name: test-app
outputRefs:
- syslogout
# ...
This configuration is compatible with both RFC3164 and RFC5424.
Example syslog message output with enrichment: None
2025-03-03T11:48:01+00:00 example-worker-x syslogsyslogserverd846bb9b: {...}
Example syslog message output with enrichment: KubernetesMinimal
2025-03-03T11:48:01+00:00 example-worker-x syslogsyslogserverd846bb9b: namespace_name=cakephp-project container_name=mysql pod_name=mysql-1-wr96h,message: {...}
12.2. Forward application logs from specific projects Copy linkLink copied to clipboard!
You can forward a copy of the application logs from specific projects to an external log aggregator, in addition to, or instead of, using the internal log store. You must also configure the external log aggregator to receive log data from OpenShift Container Platform.
To configure forwarding application logs from a project, you must create a ClusterLogForwarder custom resource (CR) with at least one input from a project, optional outputs for other log aggregators, and pipelines that use those inputs and outputs.
Prerequisites
- You must configure a logging server to receive the logging data using the specified protocol or format.
Procedure
Create or edit a YAML file that defines the
ClusterLogForwarderCR:Example
ClusterLogForwarderCRapiVersion: observability.openshift.io/v1 kind: ClusterLogForwarder metadata: name: <log_forwarder_name> namespace: <log_forwarder_namespace> spec: serviceAccount: name: <service_account_name> outputs: - name: <output_name> type: <output_type> inputs: - name: my-app-logs type: application application: includes: - namespace: my-project filters: - name: my-project-labels type: openshiftLabels openshiftLabels: project: my-project - name: cluster-labels type: openshiftLabels openshiftLabels: clusterId: C1234 pipelines: - name: <pipeline_name> inputRefs: - my-app-logs outputRefs: - <output_name> filterRefs: - my-project-labels - cluster-labelsinputs.name- Specify the name for the input.
type-
Specify the type as
applicationto collect logs from applications. includes- Specify the set of namespaces and containers to include when collecting logs.
openshiftLabels-
Specify the labels to apply to log records passing through this pipeline. These labels appear in the
openshift.labelsmap in the log record. pipelines.name- Specify a name for the pipeline.
Apply the
ClusterLogForwarderCR by running the following command:$ oc apply -f <filename>.yaml