Chapter 4. Loki query performance
Analyze and improve Loki query performance.
4.1. Analyze Loki query performance Copy linkLink copied to clipboard!
Use the Loki query performance statistics to identify and troubleshoot slow queries.
Every query and subquery in Loki generates a metrics.go log line with performance statistics. Subqueries emit the log line in the queriers. Every query has an associated single summary metrics.go line emitted by the query-front end.
Prerequisites
- You have administrator permissions.
- You have access to the OpenShift Container Platform web console.
- You installed and configured Loki Operator.
Procedure
-
In the OpenShift Container Platform web console, navigate to the Metrics
Observe tab. Note the following values:
- duration: Denotes the amount of time a query took to run.
- queue_time: Denotes the time a query spent in the queue before being processed.
- chunk_refs_fetch_time: Denotes the amount of time spent in getting chunk information from the index.
- store_chunks_download_time: Denotes the amount of time in getting chunks from cache or storage.
Calculate the following performance metrics:
total query time as
total_duration:total_duration = duration + queue_timePercentage of the total duration that a query spent in the queue as
Queue Time:Queue Time = queue_time / total_duration * 100Calculate the percentage of the total duration that was spent in getting chunk information from the index as
Chunk Refs Fetch Time:Chunk Refs Fetch Time = chunk_refs_fetch_time / total_duration * 100Calculate the percentage of the total duration that was spent in getting chunks from cache or storage:
Chunks Download Time = store_chunks_download_time / total_duration * 100Calculate the percentage of the total duration that was spent in executing the query:
Execution Time = (duration - chunk_refs_fetch_time - store_chunks_download_time) / total_duration * 100
- Refer to Query performance analysis to understand the reason for each metric and how each metric affects query performance.
4.2. Query performance analysis Copy linkLink copied to clipboard!
Learn how to analyze and optimize Loki query performance metrics.
For best query performance, you want to see as much time as possible spent in execution time, denoted by the Execution Time metric. See the table below for the reason other performance metrics might be higher and the steps you can take to improve them. You can also reduce the execution time by modifying your queries, thereby improving the overall performance.
| Issue | Reason | Fix |
|---|---|---|
|
High | Queries might be doing many CPU-intensive operations such as regular expression processing. | You can make the following changes:
|
| Your queries have many small log lines. | If your queries have many small lines, execution becomes dependent on how fast Loki can iterate the lines themselves. This becomes a CPU clock frequency bottleneck. To make things faster you need a faster CPU. | |
|
High |
You do not have enough |
The only fix is to increase the number of |
|
High |
Insufficient number of index-gateway replicas in the | Increase the number of index-gateway replicas or ensure they have enough CPU resources. |
|
High | The chunks might be too small |
Check the average chunk size by dividing |
| Query timing out | Query timeout value might be too low |
Increase the |