Chapter 6. Content distribution with Red Hat Quay
Content distribution in Red Hat Quay includes repository mirroring, geo-replication, and deployment in air-gapped environments. You can use these features to sync images across registries, replicate across regions, or run in disconnected setups.
6.1. Repository mirroring
With repository mirroring in Red Hat Quay, you can copy images from an external registry into your cluster. You sync by repository or organization name and tag, set intervals, and filter architectures.
From a Red Hat Quay registry with mirroring enabled, you can:
- Choose a repository or organization from an external registry to mirror
- Add credentials to access the external registry
- Identify specific container image repository or organization names and tags to sync
- Set intervals at which a repository or organization is synced
- Check the current state of synchronization
- Filter the architectures that are mirrored
Repository mirroring suits distinct registries that must share selected upstream content. You can configure mirroring in the Red Hat Quay configuration file or by using the Red Hat Quay API when you are ready to implement it.
6.1.1. Using mirroring
Repository mirroring in Red Hat Quay syncs images for a repository or organization from an external registry. Filters, robot accounts, and sync intervals control what is copied and when.
The following list shows features and limitations of Red Hat Quay mirroring for a repository or organization.
The word entity is used in the mirroring documentation to refer to either a repository or organization.
- With mirroring, you can mirror an entire entity or selectively limit which images are synced. Filters can be based on a comma-separated list of tags, a range of tags, or other means of identifying tags through Unix shell-style wildcards.
- After you set mirroring for an entity, you cannot manually add other images to that entity.
- Because the mirrored entity is based on the entity and the tags that you set, the entity holds only the content represented by the entity and tag pair. For example, if you change the tag so that some images in the entity no longer match, those images are deleted.
- Only the designated robot can push images to a mirrored entity, superseding any role-based access control permissions set on the entity.
- Mirroring can be configured to roll back on failure, or to run on a best-effort basis.
- With a mirrored entity, a user with read permissions can pull images from the entity but cannot push images to the entity.
- Changing settings on your mirrored entity can be performed in the Red Hat Quay user interface.
- Images are synced at set intervals, but can also be synced on demand.
In the current implementation of organization-level repository mirroring, Red Hat Quay does not replicate deletions from the source registry. If any of the following entities are removed or absent in the upstream source, they persist in the local Red Hat Quay mirror:
- Source namespaces or organizations and their repositories. If the entire upstream source namespace (for example, a Harbor project or Red Hat Quay organization) is removed, all previously mirrored repositories and their content remain in the local mirror.
- Individual repositories. Repositories previously discovered and synced continue to be tracked and served even if removed from the upstream source.
- Image tags. Tags that existed at the time of the last sync persist in the mirror even if deleted upstream.
- Referrers. OCI Referrers API artifacts, such as Cosign signatures and SBOMs, are not currently mirrored. When present locally, they are not cleaned up automatically.
The downstream mirror acts as a cumulative archive. If an upstream namespace is emptied or its repositories and tags are removed, the mirror continues to serve the last successfully synchronized versions of those objects. This might lead to higher storage consumption in the mirror than in the source.
NoteThis behavior differs from repository-level mirroring, which automatically removes local tags that are no longer present in the source registry.
Manual deletion by using the Red Hat Quay UI or API is required to remove these entities from the mirror. A future release can introduce a configurable option to automatically delete absent items, including organizations, repositories, tags, referrers, and manifest list children from the local mirror.
6.1.2. Repository mirroring capacity considerations
For Red Hat Quay repository mirroring, you can run mirroring workers on any node and size the number of workers to how many repositories you want to sync in parallel.
When you plan mirroring capacity, consider the following:
- Mirroring workers can run on nodes that already run Red Hat Quay.
- Mirroring runs in scheduled batches. More workers allow more repositories to sync at the same time. For example, 10 workers can run 10 mirroring operations in parallel.
- Size workers based on how many repositories you need to mirror in parallel, how many images and tags those repositories contain, and how often content changes.
- Multiple tags in the same repository cannot be mirrored in parallel.
6.1.3. Event notifications for mirroring
Red Hat Quay repository mirroring supports three event notifications: Mirror Started, Mirror Success, and Mirror Unsuccessful. You can configure them per repository in the Settings tab and deliver them by email, Slack, the UI, or webhooks.
The events can be configured inside of the Settings tab for each repository, and all existing notification methods such as email, Slack, Quay UI, and webhooks are supported.
6.1.4. Mirroring API
You can use the Red Hat Quay API to configure repository mirroring programmatically. The API Guide documents the available mirroring endpoints.
6.2. Geo-replication
Geo-replication connects multiple geographically distributed Red Hat Quay deployments so that clients use them as a single registry. Standalone and Operator-based deployments support geo-replication.
With geo-replication, regions share one database and one Redis instance while each region keeps local object storage. Clients push and pull through a common entrypoint, typically a global load balancer, and blob data replicates asynchronously between storage backends.
6.2.1. Geo-replication features
With geo-replication in Red Hat Quay, image pushes go to the nearest storage backend and data replicates in the background so that pulls can use the closest available engine.
The following are the key features of geo-replication:
- When geo-replication is configured, container image pushes are written to the preferred storage engine for that Red Hat Quay instance. This is typically the nearest storage backend within the region.
- After the initial push, image data is replicated in the background to other storage engines.
- The list of replication locations is configurable and those can be different storage backends.
- An image pull always uses the closest available storage engine to maximize pull performance.
- If replication has not been completed yet, the pull uses the source storage backend instead.
6.2.2. Geo-replication requirements and constraints
Red Hat Quay geo-replication requires shared storage, database, Redis, network access between regions, and a global load balancer. Review these constraints before you deploy.
The following are the requirements and constraints for geo-replication:
- In geo-replicated setups, Red Hat Quay requires that all regions are able to read and write to all other regions' object storage. Object storage must be geographically accessible by all other regions.
- Red Hat Quay has no internal awareness of object storage health. Plan a global load balancer that monitors site and storage health and redirects clients when a site must be taken offline.
- Geo-replication does not automatically fail over object storage. If one site’s storage is unavailable, you must shut down that site’s Red Hat Quay deployment so the load balancer can send clients to healthy sites. Otherwise, clients experience pull and push failures.
- Geo-replication is asynchronous. Permanent loss of a site can lose blobs that were stored there but not yet replicated.
- A single database, and therefore all metadata and Red Hat Quay configuration, is shared across all regions. Geo-replication does not replicate the database and does not fail over to another database.
- A single Redis cache is shared across the entire Red Hat Quay setup and must be accessible by all Red Hat Quay pods.
- Use the same configuration across regions, except for the preferred storage backend per region.
- Geo-replication requires object storage in each region. It does not work with local storage.
- Each region must be able to access every storage engine in each region, which requires a network path. A storage proxy is an alternative when direct paths are not practical.
- The entire storage backend, for example, all blobs, is replicated. Repository mirroring, by contrast, can be limited to a repository or an image.
- All Red Hat Quay instances must share the same entrypoint, typically through a load balancer.
- All Red Hat Quay instances must have the same set of superusers, as they are defined inside the common configuration file.
- For Operator-based geo-replication, plan a shared Clair database (typically an unmanaged Clair database) so every region uses the same vulnerability data store.
- Geo-replication requires SSL/TLS certificates and keys.
If the above requirements cannot be met, you should instead use two or more distinct Red Hat Quay deployments and take advantage of repository mirroring functions.
6.2.3. Standalone geo-replication architecture
Standalone Red Hat Quay geo-replication runs in multiple regions with a shared database and Redis and localized storage per region. Image pulls use the closest storage and pushes replicate in the background, improving performance for distributed users.
In the following image, Red Hat Quay is running standalone in two separate regions, with a common database and a common Redis instance. Localized image storage is provided in each region and image pulls are served from the closest available storage engine. Container image pushes are written to the preferred storage engine for the Red Hat Quay instance, and are then replicated in the background to the other storage engines.
If Clair fails in one cluster, for example, the US cluster, US users would not see vulnerability reports in Red Hat Quay for the second cluster (EU). This is because all Clair instances have the same state. When Clair fails, it is usually because of a problem within the cluster.
6.2.4. Operator geo-replication architecture
Red Hat Quay geo-replication on OpenShift Container Platform uses the Operator in multiple regions with a shared database and Redis and localized storage per region. Image pulls use the closest storage and pushes replicate in the background for distributed users.
In the following example, the Red Hat Quay Operator is deployed in two separate regions, with a common database and a common Redis instance. Localized image storage is provided in each region and image pulls are served from the closest available storage engine. Container image pushes are written to the preferred storage engine for the Quay instance, and are then replicated in the background to the other storage engines.
Because the Operator now manages the Clair security scanner and its database separately, geo-replication setups can be leveraged so that they do not manage the Clair database. Instead, an external shared database would be used. Red Hat Quay and Clair support several providers and vendors of PostgreSQL, which can be found in the Red Hat Quay 3.x test matrix. Additionally, the Operator also supports custom Clair configurations that can be injected into the deployment, which allows users to configure Clair with the connection credentials for the external database.
6.2.5. Mixed storage for geo-replication
Geo-replication in Red Hat Quay can use mixed storage backends, such as public-cloud object storage and on-premise Ceph, when you secure cross-site access appropriately.
Because geo-replication supports multiple replication targets, Red Hat recommends that you use a VPN or token pair with bucket-specific access to meet security requirements. This results in the public cloud instance of Red Hat Quay having access to on-premise storage, but the network is encrypted, protected, and uses ACLs, thereby meeting security requirements. If you cannot implement these security measures, it might be preferable to deploy two distinct Red Hat Quay registries and to use repository mirroring as an alternative to geo-replication.
6.3. Mirroring compared to geo-replication
Mirroring and geo-replication solve different distribution needs in Red Hat Quay. Mirroring syncs selected repositories between separate registries; geo-replication shares one database and replicates blob storage across regions.
For example, a geo-replicated Red Hat Quay registry can use two different blob storage endpoints.
The primary use cases for geo-replication include the following:
- Speeding up access to the binary blobs for geographically dispersed setups
- Guaranteeing that the image content is the same across regions
Mirroring synchronizes selected repositories, or subsets of repositories, from one registry to another. The registries are distinct, with each registry having a separate database and separate image storage.
The primary use cases for mirroring include the following:
- Independent registry deployments in different data centers or regions, where a certain subset of the overall content is supposed to be shared across the data centers and regions
- Automatic synchronization or mirroring of selected (allowlisted) upstream repositories from external registries into a local Red Hat Quay deployment
Mirroring and geo-replication can be used simultaneously.
| Feature / Capability | Geo-replication | Mirroring |
|---|---|---|
| What is the feature designed to do? | A shared, global registry | Distinct, different registries |
| What happens if replication or mirroring has not been completed yet? | The remote copy is used (slower) | No image is served |
| Is access to all storage backends in both regions required? | Yes (all Red Hat Quay nodes) | No (distinct storage) |
| Can users push images from both sites to the same repository or organization? | Yes | No |
| Is all registry content and configuration identical across all regions (shared database)? | Yes | No |
| Can users select individual namespaces or repositories to be mirrored? | No | Yes |
| Can users apply filters to synchronization rules? | No | Yes |
| Are individual / different role-based access control configurations allowed in each region | No | Yes |
6.4. Air-gapped or disconnected deployments
Red Hat Quay supports air-gapped and disconnected deployments with internet-connected registries and firewall access, or fully disconnected installations with offline media transfer. You can choose the pattern that matches your network constraints.
In the following diagram, the upper deployment in the diagram shows Red Hat Quay and Clair connected to the internet, with an air-gapped OpenShift Container Platform cluster accessing the Red Hat Quay registry through an explicit, allowlisted hole in the firewall.
The lower deployment in the diagram shows Red Hat Quay and Clair running inside of the firewall, with image and CVE data transferred to the target system using offline media. The data is exported from a separate Red Hat Quay and Clair deployment that is connected to the internet.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}