Chapter 3. Troubleshoot log forwarding

Procedure

1. Run the following query in the OpenShift web console under Observe Metrics to confirm that rate limiting is active and to discover the log tenants that the rate limit affects:

   sum by (tenant, reason) (rate(loki_discarded_samples_total[2m]))

   A nonzero result with reason="RateLimit" confirms that rate limiting is active. Monitor the metric over time to check whether the situation resolves on its own. If the result returns to zero, no further action is necessary.

2. If rate limiting persists, reduce the volume of logs sent to LokiStack before adjusting the ingestion limits:

   • Apply input rate limits to containers that produce excessive logs.
   • Use drop filters to discard log records that are not required.

   For more information, see Fine-tune log forwarding and Configure filters.

3. If reducing log volume is not enough, update the ingestionBurstSize and ingestionRate fields in the LokiStack CR:

   Note

   Increasing the ingestion rate limit increases resource usage in the LokiStack distributor pods. Before increasing these values, verify that the nodes where LokiStack pods run have enough CPU and memory available.

   To find an appropriate value for ingestionRate, run the following query in the OpenShift web console under Observe Metrics to measure the actual ingestion rate per tenant:

   sum by (tenant) (rate(loki_distributor_bytes_received_total[5m]))

   Use the result, in bytes per second, as a baseline. Convert to MB and set ingestionRate to at least that value.

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     limits:
       global:
         ingestion:
           ingestionBurstSize: 16
           ingestionRate: 8
   # ...

   spec.limits.global.ingestion.ingestionBurstSize

   The maximum local rate-limited sample size per distributor replica, in MB. This value is a hard limit. Set this value to at least the maximum size expected in a single push request. Loki rejects requests that exceed this value.

   spec.limits.global.ingestion.ingestionRate

   The maximum ingestion rate per second across all distributor replicas, in MB. This is a soft limit. When the log rate exceeds the limit, rate limit errors occur, but Vector retries the request. If the average rate is lower than the limit, the system recovers automatically.

Procedure

1. Run the following query in the OpenShift web console under Observe Metrics to confirm per-stream rate limiting is active and to discover the tenants that the per-stream rate limit affects:

   sum by (tenant, reason) (rate(loki_discarded_samples_total[2m]))

   A nonzero result with reason="StreamRateLimit" confirms that per-stream rate limiting is active.

2. Find the container producing the high log volume by reviewing the ingester pod logs for the stream labels in the error message:

   $ oc -n openshift-logging logs -l app.kubernetes.io/component=ingester | grep "Per stream rate limit exceeded"

   The error message includes labels such as kubernetes_namespace_name, kubernetes_pod_name, and kubernetes_container_name that identify the source container.

3. Reduce the volume of logs produced by the affected container:

   • Reduce the log verbosity of the application if it is running in debug or verbose mode.
   • Use drop or prune filters to discard log records that are not required before they reach LokiStack.

   For more information, see Configure filters.

4. If reducing log volume is not enough, update the perStreamRateLimit and perStreamRateLimitBurst fields in the LokiStack CR. Do not exceed the recommended maximum values:

   apiVersion: loki.grafana.com/v1
   kind: LokiStack
   metadata:
     name: logging-loki
     namespace: openshift-logging
   spec:
     limits:
       global:
         ingestion:
           perStreamRateLimit: 5
           perStreamRateLimitBurst: 20
   # ...

   Note

   The ingestionRate value must be greater than the perStreamRateLimit value.

   spec.limits.global.ingestion.perStreamRateLimit

   The maximum ingestion rate per second for a single stream, in MB. The recommended maximum value is 5.

   spec.limits.global.ingestion.perStreamRateLimitBurst

   The maximum burst size for a single stream, in MB. The recommended maximum value is 20.

Procedure

1. Run the following query in the OpenShift web console under Observe Metrics to confirm that LokiStack is discarding oversized log lines and to discover the tenants that the size limit affects:

   sum by (tenant, reason) (irate(loki_discarded_samples_total[2m]))

   A nonzero result with reason="LineTooLong" confirms that LokiStack is discarding log lines because they exceed the 256 KB limit.

2. Confirm that the collector receives 400 errors from LokiStack by reviewing the collector logs:

   $ oc -n openshift-logging logs -l app.kubernetes.io/component=collector | grep "400 Bad Request"

   Example output

   2024-11-28T17:43:30.744862692Z ERROR sink{component_kind="sink" ...}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=Server responded with an error: 400 Bad Request

   The Non-retriable error message confirms that Vector is dropping the request and the log lines are permanently lost.

3. Find the source of the oversized log lines by reviewing the distributor pod logs:

   $ for pod in $(oc get pods -o name -l app.kubernetes.io/component=distributor -n openshift-logging); \
     do oc logs $pod -n openshift-logging; \
     done | grep "Max entry size '256000' bytes exceeded for stream"

   The error message includes stream labels such as kubernetes_namespace_name, kubernetes_pod_name, and kubernetes_container_name that identify the source container.

4. Reduce the size of log lines produced by the affected container:

   • Reduce the log verbosity of the application to prevent it from producing log lines over 256 KB.
   • Change the application log format to produce shorter, structured log lines.
   • Use a prune filter to remove large fields from log records before they reach LokiStack. For more information, see Configure filters.