Chapter 13. Red Hat Quay as a proxy cache for upstream registries
You can use Red Hat Quay as a proxy cache for upstream registries to avoid pull-rate limits and speed up image pulls. Cached images update only when the upstream digest changes.
With Red Hat Quay cache proxy, the following features are available:
- Specific organizations can be defined as a cache for upstream registries.
Configuration of a Quay organization that acts as a cache for a specific upstream registry. This repository can be defined by using the Quay UI, and offers the following configurations:
- Upstream registry credentials for private repositories or increased rate limiting.
- Expiration timer to avoid surpassing cache organization size.
- Global on/off configurable via the configuration application.
-
Caching of entire upstream registries or just a single namespace, for example, all of
docker.ioor justdocker.io/library. - Logging of all cache pulls.
- Cached images scannability by Clair.
- Caching of all layers when an image is pulled from a proxied repository, which helps ensure that Clair can scan all images and that images remain pullable even if the upstream registry becomes unavailable.
13.1. Proxy cache architecture Copy linkLink copied to clipboard!
The Red Hat Quay proxy cache architecture shows how image pulls are served from cache or fetched from an upstream registry. Cache behavior depends on whether the image exists, is stale, or differs from upstream.
The following image shows the expected design flow and architecture of the proxy cache feature.
When a user pulls an image, for example, postgres:14, from an upstream repository on Red Hat Quay, the repository checks to see if an image is present. If the image does not exist, a fresh pull is initiated. After being pulled, the image layers are saved to cache and server to the user in parallel. The following image depicts an architectural overview of this scenario:
If the image in the cache exists, users can rely on Quay’s cache to stay up-to-date with the upstream source so that newer images from the cache are automatically pulled. This happens when tags of the original image have been overwritten in the upstream registry. The following image depicts an architectural overview of what happens when the upstream image and cached version of the image are different:
If the upstream image and cached version are the same, no layers are pulled and the cached image is delivered to the user.
In some cases, users initiate pulls when the upstream registry is down. If this happens with the configured staleness period, the image stored in cache is delivered. If the pull happens after the configured staleness period, the error is propagated to the user. The following image depicts an architectural overview when a pull happens after the configured staleness period:
Quay administrators can leverage the configurable size limit of an organization to limit cache size so that backend storage consumption remains predictable. This is achieved by discarding images from the cache according to the frequency in which an image is used. The following image depicts an architectural overview of this scenario:
13.2. Proxy cache limitations Copy linkLink copied to clipboard!
Proxy caching in Red Hat Quay has size, property, and anonymous-user limitations. You can use these constraints to plan cache organization capacity and access.
The following limitations apply:
- Your proxy cache must have a size limit of greater than, or equal to, the image you want to cache. For example, if your proxy cache organization has a maximum size of 500 MB, and the image a user wants to pull is 700 MB, the image will be cached and will overflow beyond the configured limit.
- Cached images must have the same properties that images on a Quay repository must have.
- Anonymous users cannot pull images through a proxy cache if the image has not been previously cached. This operation requires the creation of a new repository within the proxy organization to store the cached image, and anonymous users do not have the necessary permissions to create repositories.
13.3. Using Red Hat Quay to proxy a remote registry Copy linkLink copied to clipboard!
To proxy a remote registry with Red Hat Quay, you can configure an organization as a cache and pull images through it. You set the remote registry, optional credentials, and expiration on the UI.
Prerequisites
-
FEATURE_PROXY_CACHEin your config.yaml is set toTrue. - Assigned the Member team role. For more information about team roles, see Setting a team role by using the UI.
Procedure
- On the Red Hat Quay v2 UI, click the name of an organization, for example, cache-org.
- In the navigation pane, click Settings.
In the Remote Registry box, enter the name of the remote registry to be cached, for example,
quay.io, and click Save.NoteBy adding a namespace to the Remote Registry, for example,
quay.io/<namespace>, users in your organization will only be able to proxy from that namespace.- Optional. In the Remote Registry username box, enter the username for authenticating into the remote registry specified in the previous step. If you leave this empty, Quay will attempt to pull content anonymously from the upstream registry.
- Optional. In the Remote registry password box, enter the password for authenticating into the remote registry. If you leave this empty, Quay will attempt to pull content anonymously from the upstream registry.
Optional. Set a time in the Expiration field.
Note- The default tag Expiration field for cached images in a proxy organization is set to 86400 seconds. In the proxy organization, the tag expiration is refreshed to the value set in the UI’s Expiration field every time the tag is pulled. This feature is different than Quay’s default individual tag expiration feature. In a proxy organization, it is possible to override the individual tag feature. When this happens, the individual tag’s expiration is reset according to the Expiration field of the proxy organization.
- Expired images will disappear after the allotted time, but are still stored in Red Hat Quay. The time in which an image is completely deleted, or collected, depends on the Time Machine setting of your organization. The default time for garbage collection is 14 days unless otherwise specified.
- Optional. Check the http box if you want an unsecure protocol used. If not checked, https is used to request the remote registry.
- Click Save.
Verification
On the CLI, pull a public image from the remote registry that was specified, for example,
quay.io, acting as a proxy cache:$ podman pull <registry_url>/<organization_name>/<quayio_namespace>/<image_name>ImportantIf your organization is set up to pull from a single namespace in the remote registry, the remote registry namespace must be omitted from the URL. For example,
podman pull <registry_url>/<organization_name>/<image_name>.
13.4. Leveraging storage quota limits in proxy organizations Copy linkLink copied to clipboard!
When a proxied namespace has quota limits configured, Red Hat Quay can auto-prune the least recently used image tags so that a new push that exceeds available storage is stored instead of being skipped.
- As part of the auto-pruning feature, the tags that are marked for deletion are eventually garbage collected by the garbage collector (gc) worker process. As a result, the quota size restriction is not fully enforced during this period.
- Currently, the namespace quota size computation does not take into account the size for manifest child. This is a known issue and will be fixed in a future version of Red Hat Quay.
13.4.1. Testing the storage quota limits feature in proxy organizations Copy linkLink copied to clipboard!
Confirm that auto-pruning works in a proxy organization with storage quota limits by pulling images until the quota is exceeded and verifying that the least recently used tag is removed.
Prerequisites
- Your organization is configured to serve as a proxy organization. The following example proxies from quay.io.
-
FEATURE_PROXY_CACHEis set toTruein yourconfig.yamlfile. -
FEATURE_QUOTA_MANAGEMENTis set toTruein yourconfig.yamlfile. -
Your organization is configured with a quota limit, for example,
150 MB.
Procedure
Pull an image to your repository from your proxy organization, for example:
$ podman pull quay-server.example.com/proxytest/projectquay/quay:3.7.9Depending on the space left in your repository, you might need to pull additional images from your proxy organization, for example:
$ podman pull quay-server.example.com/proxytest/projectquay/quay:3.6.2In the Red Hat Quay registry UI, click the name of your repository.
-
Click Tags in the navigation pane and ensure that
quay:3.7.9andquay:3.6.2are tagged.
-
Click Tags in the navigation pane and ensure that
Pull the last image that will result in your repository exceeding the allotted quota, for example:
$ podman pull quay-server.example.com/proxytest/projectquay/quay:3.5.1-
Refresh the Tags page of your Red Hat Quay registry. The first image that you pushed, for example,
quay:3.7.9should have been auto-pruned. The Tags page should now showquay:3.6.2andquay:3.5.1.