Chapter 5. Fine-tune log forwarding

Procedure

1. Configure rate limits on inputs, outputs, or both.

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: collector
     inputs:
     - name: limited-app-logs
       type: application
       application:
         tuning:
           rateLimitPerContainer:
             maxRecordsPerSecond: 100
     outputs:
     - name: limited-splunk
       type: splunk
       splunk:
         url: https://splunk.example.com:8088
         authentication:
           token:
             key: token
             secretName: splunk-secret
       rateLimit:
         maxRecordsPerSecond: 1000
     pipelines:
     - name: app-to-splunk
       inputRefs:
       - limited-app-logs
       outputRefs:
       - limited-splunk

   Input rate limit (inputs[].application.tuning.rateLimitPerContainer)

   Limits the number of log records per second from each container that matches this input. This is a per-container limit. For example, if you set maxRecordsPerSecond: 100 and 10 containers match the input selector, the total input rate could be up to 1000 records per second (100 per container × 10 containers).

   Use case: Site reliability engineers can use input rate limits to prevent noisy applications from overwhelming the log collector and consuming excessive cluster resources.

   Output rate limit (outputs[].rateLimit)

   Limits the total number of log records per second that the collector forwards to this output from a single collector pod. This is an aggregate limit across all pipelines that reference this output. Generally, one collector pod runs per cluster node.

   Use case: Developers can use output rate limits to stay within the ingestion quotas imposed by third-party log aggregation services, preventing overage charges or throttling.

   Important

   When rate limits are exceeded, the collector drops log records. The collector increments the observability_log_forwarder_output_rate_limited_total metric for each dropped record. Monitor this metric to determine if your rate limits are too restrictive.

   Rate limit only on output

   spec:
     outputs:
     - name: rate-limited-kafka
       type: kafka
       kafka:
         brokers:
         - kafka.example.com:9092
         topic: logs
       rateLimit:
         maxRecordsPerSecond: 5000
     pipelines:
     - name: all-logs-to-kafka
       inputRefs:
       - application
       - infrastructure
       outputRefs:
       - rate-limited-kafka

   Rate limit only on input

   spec:
     inputs:
     - name: high-volume-namespace
       type: application
       application:
         includes:
         - namespace: "production"
         tuning:
           rateLimitPerContainer:
             maxRecordsPerSecond: 500
     pipelines:
     - name: production-to-loki
       inputRefs:
       - high-volume-namespace
       outputRefs:
       - default-lokistack

   Different rate limits for different outputs

   spec:
     outputs:
     - name: expensive-splunk
       type: splunk
       splunk:
         url: https://splunk.example.com:8088
         authentication:
           token:
             key: token
             secretName: splunk-secret
       rateLimit:
         maxRecordsPerSecond: 500
     - name: cheap-s3
       type: s3
       s3:
         url: https://s3.us-east-1.amazonaws.com
         bucket: log-archive
       rateLimit:
         maxRecordsPerSecond: 10000
     pipelines:
     - name: critical-to-splunk
       inputRefs:
       - audit
       outputRefs:
       - expensive-splunk
     - name: bulk-to-s3
       inputRefs:
       - application
       - infrastructure
       outputRefs:
       - cheap-s3

2. Apply the updated ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Set the log level by running the following command:

   $ oc annotate clusterlogforwarder instance -n openshift-logging \
     observability.openshift.io/log-level=debug --overwrite

   The supported log levels are trace, debug, info, warn, error, and off. The default level is info.

2. Verify that the collector pod restarted with the new log level by running the following command:

   $ oc logs -n openshift-logging -l app.kubernetes.io/component=collector --tail=5

   At the debug level, entries begin with DEBUG.

   Note

   Use debug or trace only for troubleshooting. These levels increase log volume significantly and can affect collector performance. Reset the level to info after troubleshooting.

Procedure

1. Patch the ClusterLogForwarder CR by running the following command:

   $ oc patch clusterlogforwarder instance -n openshift-logging \
     --type=merge \
     -p '{"spec":{"collector":{"resources":{"requests":{"cpu":"100m","memory":"256Mi"},"limits":{"cpu":"500m","memory":"512Mi"}}}}}'

2. Verify that the collector pod restarted with the updated resources by running the following command:

   $ oc get pods -n openshift-logging -l app.kubernetes.io/component=collector

   The pod age should show a recent restart.

   Note

   If you do not set resource limits explicitly, the collector applies default limits: 500m CPU and 64Mi memory for requests, and 6000m CPU and 2048Mi memory for limits. Setting explicit limits prevents the collector from consuming more resources than expected on nodes with high log volume.

Procedure

1. Create a LogFileMetricExporter CR as a YAML file:

   apiVersion: logging.openshift.io/v1alpha1
   kind: LogFileMetricExporter
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     nodeSelector: {}
     resources:
       limits:
         cpu: 500m
         memory: 256Mi
       requests:
         cpu: 200m
         memory: 128Mi
     tolerations: []

   spec.nodeSelector

   Optional: Define the nodes that the exporter pods run on.

   spec.resources

   Optional: Define CPU and memory resource limits and requests for the exporter.

   spec.tolerations

   Optional: Define tolerations that the exporter pods accept.

   Note

   The LogFileMetricExporter is a singleton. The Operator only honors openshift-logging/instance for the namespace and name.

2. Run the following command to apply the LogFileMetricExporter CR:

   $ oc apply -f <filename>.yaml