Chapter 4. Troubleshoot logging alerts

Procedure

1. Set an environment variable for your LokiStack custom resource name:

   $ LOKISTACK_NAME="logging-loki"

   Note

   This example assumes the LokiStack custom resource is named logging-loki. If your LokiStack has a different name, adjust the variable value accordingly.

2. View the LokiStack status to identify issues:

   $ oc -n openshift-logging get lokistack ${LOKISTACK_NAME} -o yaml

   Review the status.conditions section for any warnings or errors.

3. Check the detailed status of the LokiStack resource:

   $ oc -n openshift-logging describe lokistack ${LOKISTACK_NAME}

   Look for conditions with status: "False" or recent events indicating problems.

4. List all LokiStack pods and their status:

   $ oc -n openshift-logging get pods -l app.kubernetes.io/instance=${LOKISTACK_NAME}

   Verify that all component pods (distributor, ingester, querier, query-frontend, index-gateway, compactor, gateway) are in Running state.

5. Examine the logs of the affected component:

   $ oc -n openshift-logging logs <pod_name>

   Replace <pod_name> with the name of the pod from the alert labels or the pod that is experiencing issues.

6. For storage-related alerts, verify backend storage connectivity:

   1. Check if the storage secret exists and contains valid credentials:

      $ oc -n openshift-logging get secret ${LOKISTACK_NAME}-<storage_type> -o yaml

      Replace <storage_type> with your storage backend (for example, aws, azure, gcp, or s3).

   2. Review recent events for storage access errors:

      $ oc -n openshift-logging get events --sort-by=.lastTimestamp | grep -i storage

7. For rate limit alerts (LokiTenantRateLimit), check which limits are being exceeded:

   1. Navigate to the OpenShift Container Platform web console.
   2. Go to Observe > Metrics.

   3. Run the following PromQL query:

      sum by (tenant, reason) (irate(loki_discarded_samples_total[2m]))

      This shows which tenant limits are causing samples to be discarded, including the specific reason for each tenant.

8. Common remediation actions based on alert type:

   • Storage connectivity issues: Verify storage credentials, network policies, and firewall rules
   • Rate limit errors: Adjust tenant limits in the LokiStack custom resource
   • High load: Increase the number of queriers or ingesters
   • Component failures: Check pod logs and restart failing components