Chapter 3. Loki log store
Loki is an optional component of OpenShift Logging that provides local log storage for short-term troubleshooting. Loki is a horizontally scalable, highly available log aggregation system that uses labels to organize and query logs.
3.1. What Loki is
Loki is a log aggregation system inspired by Prometheus. Unlike traditional logging systems that index the full contents of every log line, Loki indexes only a small set of labels extracted from the log payload, called stream labels. The actual log content is compressed and stored in object storage such as Amazon S3, Google Cloud Storage, or Azure Blob Storage.
This label-based indexing approach makes Loki:
- Cost-effective: Loki indexes stream labels rather than the full log content, reducing indexing overhead and index storage requirements. At query time, Loki retrieves the selected chunks and filters or parses their contents as required, so broad searches can still consume substantial resources.
- Horizontally scalable: Add more ingesters or queriers as log volume grows
- Kubernetes-native: Uses Kubernetes labels you already have for pod, namespace, and container selection
3.2. How Loki uses Kubernetes labels
Loki organizes logs into streams, where each stream is identified by a unique set of labels. When Vector sends logs to Loki, it automatically extracts labels from Kubernetes metadata:
- kubernetes_namespace_name: The namespace where the pod runs
- kubernetes_pod_name: The name of the pod
- kubernetes_container_name: The container within the pod
- kubernetes_host: The node where the pod runs
- log_type: Whether the log is application, infrastructure, or audit
For infrastructure logs from systemd, Loki also extracts:
-
systemd_unit: The systemd unit that generated the log (for example,
kubelet.service)
These Kubernetes labels enable you to select and filter logs exactly as you would select pods with kubectl. For example, you can query:
- All logs from a specific namespace
- Logs from a particular pod
- All application logs across all namespaces
- Infrastructure logs from the kubelet service on specific nodes
3.3. Streams and labels
Loki stores logs as streams. Each unique combination of labels creates a separate stream. For example:
-
One stream for application logs from
pod-1innamespace-a -
A different stream for application logs from
pod-2innamespace-a -
Another stream for infrastructure logs from
kubelet.serviceonnode-1
When you query Loki by using LogQL (Loki’s query language), you first select streams by their labels, then filter the log content within those streams. This two-step process mirrors how Kubernetes works: select resources by labels, then examine their content.
3.4. How Loki works
The Loki data flow:
- Collection: Vector collects logs and extracts Kubernetes labels
- Ingestion: Loki’s distributor receives logs and forwards them to ingesters
- Temporary storage: Ingesters hold logs in memory temporarily
- Compression and storage: Ingesters compress logs into chunks and store them in object storage
- Indexing: Only the labels (metadata) are indexed, not the log content
- Querying: When you query, Loki uses the index to find relevant streams, then decompresses chunks to return matching log lines
3.5. Short-term storage design
The OpenShift Logging configuration of Loki is designed for short-term log storage and fast troubleshooting, not long-term archival.
Key characteristics:
- Tested and supported retention up to 30 days: OpenShift Logging LokiStack sizing is tested and supported for retention periods up to 30 days. Logs are not automatically expired; configure retention explicitly to ensure that logs are deleted.
- Optimized for recent queries: Performance is best when querying recent logs (last few hours or days)
- Short-term troubleshooting: Ideal for debugging current issues, not historical analysis
For long-term storage, compliance archives, or queries spanning more than 30 days, forward logs to an external log management platform in addition to or instead of using Loki.
3.6. Loki components
When you deploy Loki in OpenShift using the LokiStack custom resource, you deploy several components:
- Distributor
- Receives logs from Vector and forwards them to ingesters
- Ingester
- Temporarily holds logs in memory, then compresses and stores them as chunks in object storage
- Querier
- Fetches data from the index and object storage to answer queries
- Query Frontend
- Processes and optimizes queries before sending them to queriers
- Ruler (optional)
- Evaluates alerting and recording rules on log data
The Loki Operator manages these components and their scaling based on your LokiStack configuration.
3.7. Querying Loki with LogQL
Loki uses LogQL, a query language inspired by PromQL (Prometheus Query Language). LogQL queries have two parts:
- Label selection: Use Kubernetes labels to select streams
- Content filtering: Filter log lines by content within selected streams
Example queries:
Select all application logs from namespace
my-app:{log_type="application", kubernetes_namespace_name="my-app"}Find errors in a specific pod:
{kubernetes_namespace_name="my-app", kubernetes_pod_name="web-1"} |= "error"Query infrastructure logs from kubelet:
{log_type="infrastructure", systemd_unit="kubelet.service"}
You can query Loki through the OpenShift web console (if you have the Cluster Observability Operator installed), Grafana, or the Loki API directly.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}