Chapter 12. Prometheus and Grafana metrics under Red Hat Quay
Red Hat Quay exports a Prometheus- and Grafana-compatible metrics endpoint on each instance so that you can monitor and alert on registry activity.
12.1. Standalone Red Hat Quay
To expose Prometheus metrics for a standalone Red Hat Quay deployment, you can publish port 9091 when you start the Quay container.
Procedure
When using
podman runto start theQuaycontainer, expose the metrics port9091:$ sudo podman run -d --rm -p 80:8080 -p 443:8443 -p 9091:9091\ --name=quay \ -v $QUAY/config:/conf/stack:Z \ -v $QUAY/storage:/datastorage:Z \ registry.redhat.io/quay/quay-rhel9:v3.18.0Verify that the metrics are available:
$ curl quay.example.com:9091/metrics
12.2. Red Hat Quay Operator
To access Prometheus metrics for an Operator-managed Red Hat Quay deployment, you can use the cluster IP of the quay-metrics service.
Procedure
Determine the cluster IP for the
quay-metricsservice:$ oc get services -n quay-enterprise NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE example-registry-clair-app ClusterIP 172.30.61.161 <none> 80/TCP,8089/TCP 18h example-registry-clair-postgres ClusterIP 172.30.122.136 <none> 5432/TCP 18h example-registry-quay-app ClusterIP 172.30.72.79 <none> 443/TCP,80/TCP,8081/TCP,55443/TCP 18h example-registry-quay-config-editor ClusterIP 172.30.185.61 <none> 80/TCP 18h example-registry-quay-database ClusterIP 172.30.114.192 <none> 5432/TCP 18h example-registry-quay-metrics ClusterIP 172.30.37.76 <none> 9091/TCP 18h example-registry-quay-redis ClusterIP 172.30.157.248 <none> 6379/TCP 18hConnect to your cluster and access the metrics using the cluster IP and port for the
quay-metricsservice:$ oc debug node/master-0 sh-4.4# curl 172.30.37.76:9091/metrics # HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles. # TYPE go_gc_duration_seconds summary go_gc_duration_seconds{quantile="0"} 4.0447e-05 go_gc_duration_seconds{quantile="0.25"} 6.2203e-05 ...
12.3. Setting up Prometheus to consume metrics
To allow Prometheus to scrape metrics from every Red Hat Quay instance in a cluster, you can publish the instances under a single DNS name that Prometheus can resolve.
12.3.1. DNS configuration under Kubernetes
To provide a DNS entry for Prometheus on Kubernetes, you can configure a simple Kubernetes service that resolves to your Red Hat Quay instances.
12.3.2. DNS configuration for a manual cluster
To manage a Prometheus DNS record outside Kubernetes, you can use SkyDNS with an etcd cluster to track Red Hat Quay instance addresses.
SkyDNS can run on an etcd cluster. Entries for each Red Hat Quay instance in the cluster can be added and removed in the etcd store. SkyDNS regularly reads them from there and updates the list of Red Hat Quay instances in the DNS record accordingly.
12.4. Introduction to metrics
Red Hat Quay exposes metrics to help monitor the registry, including metrics for general registry usage, uploads, downloads, garbage collection, and authentication.
12.4.1. General registry statistics
Use these general registry statistics metrics to track how large a Red Hat Quay deployment has grown, including users, robots, organizations, repositories, and unscanned images.
| Metric name | Description |
|---|---|
| quay_user_rows | Number of users in the database |
| quay_robot_rows | Number of robot accounts in the database |
| quay_org_rows | Number of organizations in the database |
| quay_repository_rows | Number of repositories in the database |
| quay_security_scanning_unscanned_images_remaining_total | Number of images that are not scanned by the latest security scanner |
Sample metrics output
# HELP quay_user_rows number of users in the database
# TYPE quay_user_rows gauge
quay_user_rows{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="65",process_name="globalpromstats.py"} 3
# HELP quay_robot_rows number of robot accounts in the database
# TYPE quay_robot_rows gauge
quay_robot_rows{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="65",process_name="globalpromstats.py"} 2
# HELP quay_org_rows number of organizations in the database
# TYPE quay_org_rows gauge
quay_org_rows{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="65",process_name="globalpromstats.py"} 2
# HELP quay_repository_rows number of repositories in the database
# TYPE quay_repository_rows gauge
quay_repository_rows{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="65",process_name="globalpromstats.py"} 4
# HELP quay_security_scanning_unscanned_images_remaining number of images that are not scanned by the latest security scanner
# TYPE quay_security_scanning_unscanned_images_remaining gauge
quay_security_scanning_unscanned_images_remaining{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 512.4.2. Queue items
Use these queue items metrics to monitor the work queues that Red Hat Quay uses for exports, garbage collection, notifications, builds, and storage replication.
| Metric name | Description |
|---|---|
| quay_queue_items_available | Number of items in a specific queue |
| quay_queue_items_locked | Number of items that are running |
| quay_queue_items_available_unlocked | Number of items that are waiting to be processed |
Metric labels:
queue_nameThe name of the queue. One of:
exportactionlogs- Queued requests to export action logs. These logs are then processed and put in storage. A link is then sent to the requester by email.
namespacegc- Queued namespaces to be garbage collected.
notification- Queue for repository notifications to be sent out.
repositorygc- Queued repositories to be garbage collected.
secscanv4- Notification queue specific for Clair V4.
dockerfilebuild- Queue for Red Hat Quay container image builds.
imagestoragereplication- Queued blob to be replicated across multiple storages.
chunk_cleanup- Queued blob segments that need to be deleted. This is only used by some storage implementations, for example, Swift.
For example, the queue labeled repositorygc contains the repositories marked for deletion by the repository garbage collection worker. For metrics with a queue_name label of repositorygc:
-
quay_queue_items_lockedis the number of repositories currently being deleted. -
quay_queue_items_available_unlockedis the number of repositories waiting to get processed by the worker.
Sample metrics output
# HELP quay_queue_items_available number of queue items that have not expired
# TYPE quay_queue_items_available gauge
quay_queue_items_available{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="63",process_name="exportactionlogsworker.py",queue_name="exportactionlogs"} 0
...
# HELP quay_queue_items_available_unlocked number of queue items that have not expired and are not locked
# TYPE quay_queue_items_available_unlocked gauge
quay_queue_items_available_unlocked{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="63",process_name="exportactionlogsworker.py",queue_name="exportactionlogs"} 0
...
# HELP quay_queue_items_locked number of queue items that have been acquired
# TYPE quay_queue_items_locked gauge
quay_queue_items_locked{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="63",process_name="exportactionlogsworker.py",queue_name="exportactionlogs"} 012.4.3. Garbage collection metrics
Use these garbage collection metrics to track how often garbage collection workers run and how many namespaces, repositories, and blobs they remove.
| Metric name | Description |
|---|---|
| quay_gc_iterations_total | Number of iterations by the GCWorker |
| quay_gc_namespaces_purged_total | Number of namespaces purged by the NamespaceGCWorker |
| quay_gc_repos_purged_total | Number of repositories purged by the RepositoryGCWorker or NamespaceGCWorker |
| quay_gc_storage_blobs_deleted_total | Number of storage blobs deleted |
Sample metrics output
# TYPE quay_gc_iterations_created gauge
quay_gc_iterations_created{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 1.6317823190189714e+09
...
# HELP quay_gc_iterations_total number of iterations by the GCWorker
# TYPE quay_gc_iterations_total counter
quay_gc_iterations_total{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 0
...
# TYPE quay_gc_namespaces_purged_created gauge
quay_gc_namespaces_purged_created{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 1.6317823190189433e+09
...
# HELP quay_gc_namespaces_purged_total number of namespaces purged by the NamespaceGCWorker
# TYPE quay_gc_namespaces_purged_total counter
quay_gc_namespaces_purged_total{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 0
....
# TYPE quay_gc_repos_purged_created gauge
quay_gc_repos_purged_created{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 1.631782319018925e+09
...
# HELP quay_gc_repos_purged_total number of repositories purged by the RepositoryGCWorker or NamespaceGCWorker
# TYPE quay_gc_repos_purged_total counter
quay_gc_repos_purged_total{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 0
...
# TYPE quay_gc_storage_blobs_deleted_created gauge
quay_gc_storage_blobs_deleted_created{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 1.6317823190189059e+09
...
# HELP quay_gc_storage_blobs_deleted_total number of storage blobs deleted
# TYPE quay_gc_storage_blobs_deleted_total counter
quay_gc_storage_blobs_deleted_total{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 0
...12.4.4. Multipart uploads metrics
Use these multipart upload metrics to monitor blob uploads to object storage and to help identify failures when Red Hat Quay cannot complete an upload.
The multipart uploads metrics show the number of blob uploads to storage (S3, Rados, GoogleCloudStorage, RHOCS).
| Metric name | Description |
|---|---|
| quay_multipart_uploads_started_total | Number of multipart uploads to Red Hat Quay storage that started |
| quay_multipart_uploads_completed_total | Number of multipart uploads to Red Hat Quay storage that completed |
Sample metrics output
# TYPE quay_multipart_uploads_completed_created gauge
quay_multipart_uploads_completed_created{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 1.6317823308284895e+09
...
# HELP quay_multipart_uploads_completed_total number of multipart uploads to Quay storage that completed
# TYPE quay_multipart_uploads_completed_total counter
quay_multipart_uploads_completed_total{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 0
# TYPE quay_multipart_uploads_started_created gauge
quay_multipart_uploads_started_created{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 1.6317823308284352e+09
...
# HELP quay_multipart_uploads_started_total number of multipart uploads to Quay storage that started
# TYPE quay_multipart_uploads_started_total counter
quay_multipart_uploads_started_total{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="208",process_name="secscan:application"} 0
...12.4.5. Image push and pull metrics
Use these image push and pull metrics to track how many images and bytes clients upload to or download from the Red Hat Quay registry.
12.4.5.1. Image pulls total
| Metric name | Description |
|---|---|
| quay_registry_image_pulls_total | The number of images downloaded from the registry. |
Metric labels:
protocol-
The registry protocol used (should always be
v2). ref- Reference used to pull - tag, manifest.
status- HTTP return code of the request.
12.4.5.2. Image bytes pulled
| Metric name | Description |
|---|---|
| quay_registry_image_pulled_estimated_bytes_total | The number of bytes downloaded from the registry. |
Metric labels:
protocol-
The registry protocol used (should always be
v2).
12.4.5.3. Image pushes total
| Metric name | Description |
|---|---|
| quay_registry_image_pushes_total | The number of images uploaded to the registry. |
Metric labels:
protocol-
The registry protocol used (should always be
v2). pstatus- HTTP return code of the request.
pmedia_type- The uploaded manifest type.
12.4.5.4. Image bytes pushed
| Metric name | Description |
|---|---|
| quay_registry_image_pushed_bytes_total | The number of bytes uploaded to the registry. |
Sample metrics output
# HELP quay_registry_image_pushed_bytes_total number of bytes pushed to the registry
# TYPE quay_registry_image_pushed_bytes_total counter
quay_registry_image_pushed_bytes_total{host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="221",process_name="registry:application"} 0
...12.4.6. Authentication metrics
Use these authentication metrics to count registry and API authentication requests by type and by whether each request succeeded or failed.
| Metric name | Description |
|---|---|
| quay_authentication_attempts_total | Number of authentication attempts across the registry and API |
Metric labels:
auth_kindThe type of authentication used, including:
-
basic -
oauth -
credentials
-
success-
trueorfalse.
Sample metrics output
# TYPE quay_authentication_attempts_created gauge
quay_authentication_attempts_created{auth_kind="basic",host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="221",process_name="registry:application",success="True"} 1.6317843039374158e+09
...
# HELP quay_authentication_attempts_total number of authentication attempts across the registry and API
# TYPE quay_authentication_attempts_total counter
quay_authentication_attempts_total{auth_kind="basic",host="example-registry-quay-app-6df87f7b66-9tfn6",instance="",job="quay",pid="221",process_name="registry:application",success="True"} 2
...
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}