Troubleshooting for Critical Alerts

Check the Elasticsearch cluster health and verify that the cluster status is red.

oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- health

List the nodes that have joined the cluster.

oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cat/nodes?v

List the Elasticsearch pods and compare them with the nodes in the command output from the previous step.

oc -n openshift-logging get pods -l component=elasticsearch

If some of the Elasticsearch nodes have not joined the cluster, perform the following steps.

If all the nodes have joined the cluster, perform the following steps, check if the cluster is in the process of recovering.

oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cat/recovery?active_only=true

Check if there are pending tasks.

oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- health |grep  number_of_pending_tasks

If there are pending tasks, monitor their status.

If it seems like the recovery has stalled, check if cluster.routing.allocation.enable is set to none.

oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cluster/settings?pretty

If cluster.routing.allocation.enable is set to none, set it to all.

oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cluster/settings?pretty -X PUT -d '{"persistent": {"cluster.routing.allocation.enable":"all"}}'

Check which indices are still red.

oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cat/indices?v

If any indices are still red, try to clear them by performing the following steps.

If the preceding steps do not clear the red indices, delete the indices individually.

If there are no red indices and the cluster status is red, check for a continuous heavy processing load on a data node.

Increase the node count by adjusting nodeCount in the ClusterLogging CR.

Identify the node on which Elasticsearch is deployed.

oc -n openshift-logging get po -o wide

Check if there are unassigned shards.

oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cluster/health?pretty | grep unassigned_shards

If there are unassigned shards, check the disk space on each node.

for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod -- df -h /elasticsearch/persistent; done

Check the nodes.node_name.fs field to determine the free disk space on that node.

Try to increase the disk space on all nodes.

If increasing the disk space is not possible, try adding a new data node to the cluster.

If adding a new data node is problematic, decrease the total cluster redundancy policy.

If the preceding steps do not fix the issue, delete the old indices.

Identify the node on which Elasticsearch is deployed.

oc -n openshift-logging get po -o wide

Check the disk space on each node.

for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod -- df -h /elasticsearch/persistent; done

Check if the cluster is rebalancing.

oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_cluster/health?pretty | grep relocating_shards

To allocate shards to a particular node, free up some space.

Try to increase the disk space on all nodes.

If increasing the disk space is not possible, try adding a new data node to the cluster.

If adding a new data node is problematic, decrease the total cluster redundancy policy.

If the preceding steps do not fix the issue, delete the old indices.

Check the disk space of the Elasticsearch node.

for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod -- df -h /elasticsearch/persistent; done

If the used disk percentage is above 95%, it signifies that the node has crossed the flood watermark. Writing is blocked for shards allocated on this particular node.

Try to increase the disk space on all nodes.

If increasing the disk space is not possible, try adding a new data node to the cluster.

If adding a new data node is problematic, decrease the total cluster redundancy policy.

If the preceding steps do not fix the issue, delete the old indices.

Continue freeing up and monitoring the disk space until the used disk space drops below 90%. Then, unblock write to this particular node.

oc exec -n openshift-logging -c elasticsearch <elasticsearch_pod_name> -- es_util --query=_all/_settings?pretty -X PUT -d '{"index.blocks.read_only_allow_delete": null}'

Get the disk space of the Elasticsearch node.

for pod in `oc -n openshift-logging get po -l component=elasticsearch -o jsonpath='{.items[*].metadata.name}'`; do echo $pod; oc -n openshift-logging exec -c elasticsearch $pod -- df -h /elasticsearch/persistent; done

In the command output, check the nodes.node_name.fs field to determine the free disk space on that node.

Try to increase the disk space on all nodes.

If increasing the disk space is not possible, try adding a new data node to the cluster.

If adding a new data node is problematic, decrease the total cluster redundancy policy.

If the preceding steps do not fix the issue, delete the old indices.