Chapter 2. Scale LokiStack deployments

You can adjust LokiStack capacity by changing the overall t-shirt size or by scaling individual component replicas. Understanding when to use each approach helps optimize resources and maintain performance.

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  size: 1x.small  # Change from 1x.extra-small

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  size: 1x.small
  template:
    querier:
      replicas: 4  # Override default for 1x.small (normally 2)
    queryFrontend:
      replicas: 3  # Override default for 1x.small (normally 2)

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
  name: logging-loki
  namespace: openshift-logging
spec:
  size: 1x.small
  template:
    distributor:
      replicas: 4  # Override default for 1x.small (normally 2)
    ingester:
      replicas: 4  # Override default for 1x.small (normally 2)

Enable zone-aware data replication to protect against log loss during zone failures.

The Loki Operator offers support for zone-aware data replication through pod topology spread constraints. Enabling this feature enhances reliability and safeguards against log loss if a single zone failure occurs. When configuring the deployment size as 1x.extra-small, 1x.small, or 1x.medium, the replication.factor field is automatically set to 2.

To ensure proper replication, you need to have at least as many availability zones as the replication factor specifies. While it is possible to have more availability zones than the replication factor, having fewer zones can lead to write failures. Each zone should host an equal number of instances for optimal operation.

apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
 name: logging-loki
 namespace: openshift-logging
spec:
 replicationFactor: 2
 replication:
   factor: 2
   zones:
   -  maxSkew: 1
      topologyKey: topology.kubernetes.io/zone

Manually delete Loki persistent volumes in failed zones to allow automatic rescheduling.

In OpenShift Container Platform a zone failure happens when specific availability zone resources become inaccessible. Availability zones are isolated areas within a cloud provider’s data center, aimed at enhancing redundancy and fault tolerance. If your OpenShift Container Platform cluster is not configured to handle this, a zone failure can lead to service or data loss.

Loki pods are part of a StatefulSet, and they come with Persistent Volume Claims (PVCs) provisioned by a StorageClass object. Each Loki pod and its PVCs reside in the same zone. When a zone failure occurs in a cluster, the StatefulSet controller automatically attempts to recover the affected pods in the failed zone.

The StatefulSet controller automatically attempts to reschedule pods in a failed zone. Because the associated PVCs are also in the failed zone, automatic rescheduling to a different zone does not work. You must manually delete the PVCs in the failed zone to allow successful re-creation of the stateful Loki Pod and its provisioned PVC in the new zone.

$ oc patch pvc <pvc_name> -p '{"metadata":{"finalizers":null}}' -n openshift-logging