Chapter 1. Troubleshoot the installation
Diagnose and resolve common installation issues. Most installation failures occur when cluster prerequisites are not met before installing operators.
Missing storage prerequisites cause many silent installation failures. Verify that your cluster meets both storage requirements before installing operators:
-
Block storage (StorageClass): For
LokiStackinternal PVCs - Object storage (S3-compatible): For actual log data
Verify that your cluster meets both storage requirements before installing operators.
1.1. Troubleshoot installation symptoms Copy linkLink copied to clipboard!
Use this reference to diagnose common installation issues. Many installation failures occur when cluster prerequisites are not met before operator installation.
- ResolutionFailed on the Loki Operator subscription
Symptom: The Loki Operator subscription shows
ResolutionFailedstatus.Cause: Incorrect channel name or source catalog.
Resolution:
Verify available channels by running the following command:
$ oc get packagemanifest loki-operator -n openshift-marketplace-
Ensure the subscription uses
source: redhat-operators(not the community catalog).
- LokiStack pods stuck in Pending state
Symptom: After creating the
LokiStackcustom resource, pods remain inPendingstate and never start.Cause: No
StorageClassexists, or the specifiedStorageClassdoes not support dynamic provisioning.Common on: single-node OpenShift deployments and edge clusters.
Resolution:
Verify that a
StorageClassexists by running the following command:$ oc get storageclassIf no
StorageClassis listed, you must configure storage beforeLokiStackcan deploy.- For single-node OpenShift or edge deployments without dynamic storage, see "Configuring local storage for single-node and edge deployments".
- For multi-node clusters, see "Verifying StorageClass availability".
- LokiStack shows Ready but logs are not stored
Symptom: The
LokiStackcustom resource showsReady: True, operators are installed, but logs disappear and are not stored or queryable.Cause: Object storage is not configured or the object storage secret is incorrect.
Resolution:
LokiStackrequires two separate types of storage:- Block storage (StorageClass): For internal PVCs
- Object storage (S3-compatible): For actual log data
See "Configuring object storage for LokiStack" to configure the S3-compatible backend and create the storage secret.
- Logs are silently dropped with no error
Symptom: The
LokiStackshowsReady: True, operators are installed, log collector pods are running, but logs are silently dropped and never appear in queries.Cause: Missing TLS CA bundle for object storage, or incorrect object storage permissions.
Resolution:
- If using self-signed certificates or private CA for object storage, see "Configuring object storage for LokiStack" to configure the CA bundle.
- Verify that the object storage credentials have all required S3 permissions. See "Object storage permissions requirements".
- Certificate verify failed error in collector logs
Symptom: The
ClusterLogForwardercustom resource showsReady: True, but no logs appear inLokiStackor other HTTP-based sinks. The collector logs show the following error:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2123:: self-signed certificate in certificate chainCause: The
tls.cablock is missing or incorrect in theClusterLogForwarderoutput configuration.Resolution:
Add the
tls.cablock to yourClusterLogForwarderoutput:spec: outputs: - name: default-lokistack type: lokiStack tls: ca: configMapName: logging-loki-gateway-ca-bundle key: service-ca.crtThe Loki Operator creates the
logging-loki-gateway-ca-bundleConfigMapautomatically. For self-signed certificates on external outputs, reference your custom CAConfigMap.- 429 Too Many Requests in collector logs
Symptom: The collector logs show
429 Too Many Requestsresponses and the collector retries requests.Cause: The collector is sending logs faster than the
LokiStackor destination sink can process them.Common during:
- Initial deployment when the collector flushes a backlog of accumulated logs
-
After restoring connectivity to a
LokiStackthat was temporarily unavailable
Resolution:
This condition resolves automatically. The collector retries with exponential back-off. On a
1x.demoLokiStack, the backlog typically clears within several minutes. LargerLokiStacksizes clear backlogs faster.- No Logs tab under Observe
Symptom: After installing operators and creating
LokiStack, the Logs tab does not appear under Observe in the web console.Cause: The Cluster Observability Operator is not installed, or the
UIPlugincustom resource was not created.Resolution:
Both steps are required to enable the web console logs interface:
- See "Install the Cluster Observability Operator".
-
After the operator is installed, create the
UIPlugincustom resource as described in the installation procedure.
- ClusterLogForwarder Ready but no logs appear
Symptom: The
ClusterLogForwarderstatus showsReady: Truebut no logs appear inLokiStackor destination sinks.Cause: The
ClusterLogForwardervalidation passes even when the collector cannot connect toLokiStack. The most common cause is missing TLS CA configuration.Resolution:
Check the collector pod logs by running the following command:
$ oc logs -n openshift-logging -l app.kubernetes.io/component=collector --tail=50- Look for TLS certificate errors or connection failures.
-
If you see certificate errors, add the
tls.cablock to yourClusterLogForwarderoutput as described in "Certificate verify failed error in collector logs."