Chapter 5. Log forwarding concepts
Log forwarding uses inputs, outputs, filters, and pipelines defined in the ClusterLogForwarder CR to route logs from your cluster to storage destinations. This section covers the general forwarding mechanism and how to set up the ClusterLogForwarder resource. For destination-specific configuration procedures such as Splunk, CloudWatch, or Kafka, see the Integrate section.
5.1. Log forwarding
You can design a log routing pipeline that connects raw cluster telemetry to the specific needs of your organization. A managed data service can help you balance deep operational visibility for site reliability engineers (SREs) with strict long-term retention requirements for security and compliance officers.
In a complex OpenShift Container Platform environment, a one-size-fits-all approach to logging is often too expensive or legally insufficient. Log forwarding helps you do the following:
- Reduce costs: Use filters to drop "noisy" debug logs or rate-limit high-volume applications before they hit expensive storage.
- Ensure compliance: Isolate sensitive audit trails and route them to immutable "cold storage" for long-term legal retention.
- Empower teams: Deliver application-specific logs directly to tools such as LokiStack, Splunk, or Datadog, allowing developers to use familiar platforms they already know.
By orchestrating multiple log routing pipelines, you can resolve the natural tensions between different log consumers:
- Cost versus visibility: Provide SREs with high-volume logs for short-term troubleshooting without incurring the costs of long-term storage on high-performance systems.
- Compliance versus accessibility: Create a dedicated audit log routing pipeline that bypasses standard developer tools to ship raw data directly to an immutable security archive.
- Data sovereignty: Ensure that filters prune specific fields or logs from sensitive namespaces before they leave the cluster for a third-party SaaS provider.
5.2. How log forwarding works
Log forwarding is powered by a collector that runs as a DaemonSet on every node in your cluster. To manage how this collector handles data, you define a ClusterLogForwarder custom resource (CR) that routes logs through three distinct functional areas:
- Inputs
Inputs define the source of your telemetry. The collector recognizes three reserved, high-level input types:
- Application: Application logs provide telemetry for user-deployed workloads.
- Infrastructure: Infrastructure logs provide telemetry for the node operating system and OpenShift Container Platform components.
Audit: Audit logs provide telemetry for sensitive security and API server records.
In addition to these default input types, you can configure receiver inputs that allow the collector to listen for incoming log data from external sources. Receiver inputs enable the collector to act as a server that accepts logs over HTTP or syslog protocols. This is useful when logs originate from Red Hat-supported products installed on the same cluster, such as Red Hat OpenStack Services on OpenShift (RHOSO) or OpenShift Virtualization, or when logging is installed on hosted control planes.
TipYou can also define custom inputs to target specific namespaces or pods using label selectors, which can isolate data for specific application teams.
- Filters
-
Filters serve as the logic gates where you enforce data policies. You use filters to refine the stream before it leaves the cluster. This includes using a
prunefilter to remove specific metadata or JSON fields containing personally identifiable information (PII), and adropfilter to discard records based on regular expressions or severity levels. The collector also supports rate limiting to prevent a single high-volume application from overwhelming your storage infrastructure. - Outputs
- Outputs specify the final destination for the telemetry. You can configure multiple outputs to satisfy different consumers simultaneously. Supported destinations include internal stores such as LokiStack, external providers such as Splunk, Datadog, or Amazon CloudWatch, and standard protocols such as Kafka, Syslog, or S3-compatible object storage.
Log routing pipelines act as the central orchestrator of the entire log routing process. A pipeline is not a step in the sequence, but the configuration logic that binds inputs and filters to specific outputs. By defining multiple pipelines within a single ClusterLogForwarder CR, you can split a single source into different paths. For example, one log routing pipeline can send raw audit logs to an S3 bucket for compliance, while a second pipeline filters those same logs for errors and sends them to a site reliability engineer (SRE) alerting dashboard.
Service account permissions ensure the collector has the necessary authority to gather logs. The collector uses a specific service account, identified in the spec.serviceAccount.name field of the CR. You must provide this service account with the appropriate ClusterRoleBinding permissions for each log type that you intend to collect.
5.3. Collector permissions and identity
The Red Hat OpenShift Logging Operator uses a collector to gather logs from across your cluster. For the collector to access these logs, it must have a distinct role and explicit permission to read specific types of log data from the nodes and the API.
- Service account
-
When you configure log forwarding, you must first create a
ServiceAccountin theopenshift-loggingnamespace. This service account acts as the identity for the collector pods. You must reference this account by name when you create yourClusterLogForwardercustom resource (CR). - Log types and cluster roles
OpenShift Container Platform protects log data by using role-based access control (RBAC). To allow the collector to "see" logs, you must create
ClusterRoleBindingsthat connect your service account to the following predefined roles:-
collect-application-logs: Grants access to logs from user-defined projects. -
collect-infrastructure-logs: Grants access to logs from the nodes and system projects such asopenshift-*andkube-*. -
collect-audit-logs: Grants access to the sensitive audit logs that the API server and OAuth server generate.
-
Sequence is critical. You must grant these permissions before you include the corresponding log types in your ClusterLogForwarder configuration.
If the ClusterLogForwarder CR requests a log type that the collector’s service account is not authorized to access, the Operator enters a protective failure state. To prevent an insecure or partial configuration, the Operator removes the entire collector DaemonSet, which stops all log collection for the cluster.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}