Chapter 4. Loki query performance


Analyze and improve Loki query performance.

4.1. Analyze Loki query performance

Use the Loki query performance statistics to identify and troubleshoot slow queries.

Every query and subquery in Loki generates a metrics.go log line with performance statistics. Subqueries emit the log line in the queriers. Every query has an associated single summary metrics.go line emitted by the query-front end.

Prerequisites

  • You have administrator permissions.
  • You have access to the OpenShift Container Platform web console.
  • You installed and configured Loki Operator.

Procedure

  1. In the OpenShift Container Platform web console, navigate to the Metrics Observe tab.
  2. Note the following values:

    • duration: Denotes the amount of time a query took to run.
    • queue_time: Denotes the time a query spent in the queue before being processed.
    • chunk_refs_fetch_time: Denotes the amount of time spent in getting chunk information from the index.
    • store_chunks_download_time: Denotes the amount of time in getting chunks from cache or storage.
  3. Calculate the following performance metrics:

    • total query time as total_duration:

      total_duration = duration + queue_time
    • Percentage of the total duration that a query spent in the queue as Queue Time:

      Queue Time = queue_time / total_duration * 100
    • Calculate the percentage of the total duration that was spent in getting chunk information from the index as Chunk Refs Fetch Time:

      Chunk Refs Fetch Time = chunk_refs_fetch_time / total_duration * 100
    • Calculate the percentage of the total duration that was spent in getting chunks from cache or storage:

      Chunks Download Time = store_chunks_download_time / total_duration * 100
    • Calculate the percentage of the total duration that was spent in executing the query:

      Execution Time = (duration - chunk_refs_fetch_time - store_chunks_download_time) / total_duration * 100
  4. Refer to Query performance analysis to understand the reason for each metric and how each metric affects query performance.

4.2. Query performance analysis

Learn how to analyze and optimize Loki query performance metrics.

For best query performance, you want to see as much time as possible spent in execution time, denoted by the Execution Time metric. See the table below for the reason other performance metrics might be higher and the steps you can take to improve them. You can also reduce the execution time by modifying your queries, thereby improving the overall performance.

IssueReasonFix

High Execution Time

Queries might be doing many CPU-intensive operations such as regular expression processing.

You can make the following changes:

  • Change your queries to reduce or remove regular expressions.
  • Add more CPU resources.

Your queries have many small log lines.

If your queries have many small lines, execution becomes dependent on how fast Loki can iterate the lines themselves. This becomes a CPU clock frequency bottleneck. To make things faster you need a faster CPU.

High Queue Time

You do not have enough queriers running.

The only fix is to increase the number of queriers replicas in the LokiStack spec.

High Chunk Refs Fetch Time

Insufficient number of index-gateway replicas in the LokiStack spec.

Increase the number of index-gateway replicas or ensure they have enough CPU resources.

High Chunks Download Time

The chunks might be too small

Check the average chunk size by dividing total_bytes value by cache_chunk_req value. The average represents the average uncompressed bytes per chunk. The value for best performance should be in the order of magnitude of megabytes. If the chunks are only a few hundred bytes or KB in size, revisit labels to ensure that you are not splitting your data into very small chunks.

Query timing out

Query timeout value might be too low

Increase the queryTimeout value in the LokiStack spec.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

Legal Notice

Theme

© 2026 Red Hat
Back to top