SupportConsoleDevelopersStart a trialContact

Chapter 16. Backing up and restoring Red Hat Quay on a standalone deployment

You can back up and restore a standalone Red Hat Quay deployment, including optional read-only mode during maintenance.

16.1. Enabling read-only mode for Red Hat Quay
Copy link

You can enable read-only mode for a standalone Red Hat Quay deployment to restrict writes during maintenance while the registry continues to serve images.

Red Hat Quay administrators can enable read-only mode to restrict write access to the registry, which helps ensure data integrity, mitigate risks during maintenance windows, and provide a safeguard against unintended modifications to registry data. Read-only mode also helps keep your Red Hat Quay registry online and available to serve images to users.

Note

In some cases, you cannot use a read-only option for Red Hat Quay because read-only mode requires inserting a service key and other manual configuration changes. As an alternative to read-only mode, Red Hat Quay administrators might consider enabling the DISABLE_PUSHES feature. When this field is set to true, users are unable to push images or image tags to the registry when using the CLI. Enabling DISABLE_PUSHES differs from read-only mode because the database is not set as read-only when that feature is enabled.

This field might be useful in some situations such as when Red Hat Quay administrators want to calculate their registry’s quota and disable image pushing until after calculation has completed. With this method, administrators can avoid putting the whole registry in read-only mode, which affects the database, so that most operations can still be done.

To create service keys for standalone Red Hat Quay read-only mode, you can generate a key pair inside the Quay container or in a local Python virtual environment.

Red Hat Quay uses service keys to communicate with various components. These keys are used to sign completed requests, such as requesting to scan images, login, storage access, and so on.

Prerequisites

  • If you are using Red Hat Enterprise Linux (RHEL) 7.x:

    • You have enabled the Red Hat Software Collections List (RHSCL).
    • You have installed Python 3.6.
    • You have downloaded the virtualenv package.
    • You have installed the git CLI.

  • If you are using Red Hat Enterprise Linux (RHEL) 8:

    • You have installed Python 3 on your machine.
    • You have downloaded the python3-virtualenv package.
    • You have installed the git CLI.
  • You have cloned the quay/quay repository from GitHub.

Procedure

  1. If your Red Hat Quay registry is readily available, you can generate service keys inside of the Quay registry container.

    1. Enter the following command to generate a key pair inside of the Quay container: 

      $ podman exec quay python3 tools/generatekeypair.py quay-readonly

  2. If your Red Hat Quay is not readily available, you must generate your service keys inside of a virtual environment.

    1. Change into the directory of your Red Hat Quay deployment and create a virtual environment inside of that directory: 

      $ cd <$QUAY>/quay && virtualenv -v venv

    2. Activate the virtual environment by entering the following command: 

      $ source venv/bin/activate

    3. Optional. Install the pip CLI tool if you do not have it installed: 

      $ venv/bin/pip install --upgrade pip

    4. In your Red Hat Quay directory, create a requirements-generatekeys.txt file with the following content: 

      $ cat << EOF > requirements-generatekeys.txt
cryptography==3.4.7
pycparser==2.19
pycryptodome==3.9.4
pycryptodomex==3.9.4
pyjwkest==1.4.2
PyJWT==1.7.1
Authlib==1.0.0a2
EOF

    5. Enter the following command to install the Python dependencies defined in the requirements-generatekeys.txt file: 

      $ venv/bin/pip install -r requirements-generatekeys.txt

    6. Enter the following command to create the necessary service keys: 

      $ PYTHONPATH=. venv/bin/python /<path_to_cloned_repo>/tools/generatekeypair.py quay-readonly

      Example output:

      Writing public key to quay-readonly.jwk
Writing key ID to quay-readonly.kid
Writing private key to quay-readonly.pem

    7. Enter the following command to deactivate the virtual environment: 

      $ deactivate

16.1.2. Adding keys to the PostgreSQL database
Copy link

To register read-only service keys with Red Hat Quay, you can insert the key and approval records into the PostgreSQL database.

Prerequisites

  • You have created the service keys.

Procedure

  1. Enter the following command to enter your Red Hat Quay database environment: 

    $ podman exec -it postgresql-quay psql -U postgres -d quay

  2. Display the approval types and associated notes of the servicekeyapproval by entering the following command: 

    quay=# select * from servicekeyapproval;

    Example output:

     id | approver_id |          approval_type           |       approved_date        | notes
----+-------------+----------------------------------+----------------------------+-------
  1 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:48.181347 |
  2 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:47:55.808087 |
  3 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:04.27095  |
  4 |             | ServiceKeyApprovalType.AUTOMATIC | 2024-05-07 03:49:05.46235  |
  5 |           1 | ServiceKeyApprovalType.SUPERUSER | 2024-05-07 04:05:10.296796 |
...

  3. Add the service key to your Red Hat Quay database by entering the following query: 

    quay=# INSERT INTO servicekey
  (name, service, metadata, kid, jwk, created_date, expiration_date)
  VALUES ('quay-readonly',
           'quay',
           '{}',
           '<contents_of_.kid_file>',
           '<contents_of_.jwk_file>',
           '<created_date_of_read-only>',
           '<expiration_date_of_read-only>');

    Example output:

    INSERT 0 1

  4. Next, add the key approval with the following query: 

    quay=# INSERT INTO servicekeyapproval ('approval_type', 'approved_date', 'notes')
  VALUES ("ServiceKeyApprovalType.SUPERUSER", "CURRENT_DATE",
           <include_notes_here_on_why_this_is_being_added>);

    Example output:

    INSERT 0 1

  5. Set the approval_id field on the created service key row to the id field from the created service key approval. You can use the following SELECT statements to get the necessary IDs: 

    UPDATE servicekey
SET approval_id = (SELECT id FROM servicekeyapproval WHERE approval_type = 'ServiceKeyApprovalType.SUPERUSER')
WHERE name = 'quay-readonly';
     
    UPDATE 1

To put a standalone Red Hat Quay deployment into read-only mode, you can add the service key files and REGISTRY_STATE settings to your configuration bundle and restart Quay.

After the service keys have been created and added to your PostgreSQL database, you must restart the Quay container on your standalone deployment.

Prerequisites

  • You have created the service keys and added them to your PostgreSQL database.

Procedure

  1. Shut down all Red Hat Quay instances on all virtual machines. For example: 

    $ podman stop <quay_container_name_on_virtual_machine_a>
     
    $ podman stop <quay_container_name_on_virtual_machine_b>

  2. Enter the following command to copy the contents of the quay-readonly.kid file and the quay-readonly.pem file to the directory that holds your Red Hat Quay configuration bundle: 

    $ cp quay-readonly.kid quay-readonly.pem $Quay/config

  3. Enter the following command to set file permissions on all files in your configuration bundle folder: 

    $ setfacl -m user:1001:rw $Quay/config/*

  4. Modify your Red Hat Quay config.yaml file and add the following information: 

    # ...
REGISTRY_STATE: readonly
INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
# ...
  5. Distribute the new configuration bundle to all Red Hat Quay instances.

  6. Start Red Hat Quay by entering the following command: 

    $ podman run -d --rm -p 80:8080 -p 443:8443  \
   --name=quay-main-app \
   -v $QUAY/config:/conf/stack:Z \
   -v $QUAY/storage:/datastorage:Z \
   {productrepo}/{quayimage}:{productminv}

  7. After starting Red Hat Quay, a banner inside your instance informs users that Red Hat Quay is running in read-only mode. Pushes should be rejected and a 405 error should be logged. You can test this by running the following command: 

    $ podman push <quay-server.example.com>/quayadmin/busybox:test

    Example output:

    613be09ab3c0: Preparing
denied: System is currently read-only. Pulls will succeed but all write operations are currently suspended.

    With your Red Hat Quay deployment on read-only mode, you can safely manage your registry’s operations and perform such actions as backup and restore.

  8. Optional. After you finish with read-only mode, you can return to normal operations by removing the following information from your config.yaml file. Then, restart your Red Hat Quay deployment: 

    # ...
REGISTRY_STATE: readonly
INSTANCE_SERVICE_KEY_KID_LOCATION: 'conf/stack/quay-readonly.kid'
INSTANCE_SERVICE_KEY_LOCATION: 'conf/stack/quay-readonly.pem'
# ...
     
    $ podman restart <container_id>

16.1.4. Updating read-only expiration time
Copy link

To extend the lifetime of a Red Hat Quay read-only service key, you can update the key expiration date in the PostgreSQL database.

The Red Hat Quay read-only key has an expiration date, and when that date passes the key is deactivated. Before the key expires, you can update its expiration time in the database.

Procedure

  1. Connect to your Red Hat Quay production database by using the methods described earlier.

  2. Optional. List service key IDs by running the following query: 

    SELECT id, name, expiration_date FROM servicekey;

  3. Update the key expiration by issuing the following query: 

    quay=# UPDATE servicekey SET expiration_date = 'new-date' WHERE id = servicekey_id;

To back up a standalone Red Hat Quay deployment, you can archive configuration files, dump the PostgreSQL database, and sync object storage blobs.

Procedure

  1. Create a temporary backup directory, for example, quay-backup: 

    $ mkdir /tmp/quay-backup

  2. The following example command denotes the local directory that the Red Hat Quay was started in, for example, /opt/quay-install: 

    $ podman run --name quay-app \
   -v /opt/quay-install/config:/conf/stack:Z \
   -v /opt/quay-install/storage:/datastorage:Z \
   registry.redhat.io/quay/quay-rhel9:v3.18.0

    Change into the directory that bind-mounts to /conf/stack inside of the container, for example, /opt/quay-install, by running the following command: 

    $ cd /opt/quay-install

  3. Compress the contents of your Red Hat Quay deployment into an archive in the quay-backup directory by entering the following command: 

    $ tar cvf /tmp/quay-backup/quay-backup.tar.gz *

    Example output:

    config.yaml
config.yaml.bak
extra_ca_certs/
extra_ca_certs/ca.crt
ssl.cert
ssl.key

  4. Back up the Quay container service by entering the following command: 

    $ podman inspect quay-app | jq -r '.[0].Config.CreateCommand | .[]' | paste -s -d ' ' -

  /usr/bin/podman run --name quay-app \
  -v /opt/quay-install/config:/conf/stack:Z \
  -v /opt/quay-install/storage:/datastorage:Z \
  registry.redhat.io/quay/quay-rhel9:v3.18.0

  5. Redirect the contents of your conf/stack/config.yaml file to your temporary quay-config.yaml file by entering the following command: 

    $ podman exec -it quay cat /conf/stack/config.yaml > /tmp/quay-backup/quay-config.yaml

  6. Obtain the DB_URI located in your temporary quay-config.yaml by entering the following command: 

    $ grep DB_URI /tmp/quay-backup/quay-config.yaml

    Example output:

    $ postgresql://<username>:test123@172.24.10.50/quay

  7. Extract the PostgreSQL contents to your temporary backup directory in a backup .sql file by entering the following command: 

    $ pg_dump -h 172.24.10.50  -p 5432 -d quay  -U  <username>   -W -O > /tmp/quay-backup/quay-backup.sql

  8. Print the contents of your DISTRIBUTED_STORAGE_CONFIG by entering the following command: 

    DISTRIBUTED_STORAGE_CONFIG:
   default:
    - S3Storage
    - s3_bucket: <bucket_name>
      storage_path: /registry
      s3_access_key: <s3_access_key>
      s3_secret_key: <s3_secret_key>
      host: <host_name>
      s3_region: <region>

  9. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 7: 

    $ export AWS_ACCESS_KEY_ID=<access_key>

  10. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 7: 

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

  11. Sync the quay bucket to the /tmp/quay-backup/blob-backup/ directory from the hostname of your DISTRIBUTED_STORAGE_CONFIG: 

    $ aws s3 sync s3://<bucket_name>  /tmp/quay-backup/blob-backup/ --source-region us-east-2

    Example output:

    download: s3://<user_name>/registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a to registry/sha256/9c/9c3181779a868e09698b567a3c42f3744584ddb1398efe2c4ba569a99b823f7a
download: s3://<user_name>/registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d to registry/sha256/e9/e9c5463f15f0fd62df3898b36ace8d15386a6813ffb470f332698ecb34af5b0d

    Note

    Delete the quay-config.yaml file after syncing the quay bucket because that file contains sensitive information. The quay-config.yaml file remains available in the quay-backup.tar.gz archive.

To restore a standalone Red Hat Quay deployment from backup, you can restore configuration files, recreate the PostgreSQL database, and sync blobs to object storage.

Prerequisites

  • You have backed up your Red Hat Quay deployment.

Procedure

  1. Create a new directory that bind-mounts to /conf/stack inside of the Red Hat Quay container: 

    $ mkdir /opt/new-quay-install

  2. Copy the contents of your temporary backup directory created in the backup procedure to the new-quay-install directory created in Step 1: 

    $ cp /tmp/quay-backup/quay-backup.tar.gz /opt/new-quay-install/

  3. Change into the new-quay-install directory by entering the following command: 

    $ cd /opt/new-quay-install/

  4. Extract the contents of your Red Hat Quay directory: 

    $ tar xvf /tmp/quay-backup/quay-backup.tar.gz *

    Example output:

    config.yaml
config.yaml.bak
extra_ca_certs/
extra_ca_certs/ca.crt
ssl.cert
ssl.key

  5. Recall the DB_URI from your backed-up config.yaml file by entering the following command: 

    $ grep DB_URI config.yaml

    Example output:

    postgresql://<username>:test123@172.24.10.50/quay

  6. Run the following command to enter the PostgreSQL database server: 

    $ sudo postgres

  7. Enter psql and create a new database in 172.24.10.50 to restore the quay databases, for example, example_restore_registry_quay_database, by entering the following command: 

    $ psql "host=172.24.10.50  port=5432 dbname=postgres user=<username>  password=test123"
postgres=> CREATE DATABASE example_restore_registry_quay_database;

    Example output:

    CREATE DATABASE

  8. Connect to the database by running the following command: 

    postgres=# \c "example-restore-registry-quay-database";

    Example output:

    You are now connected to database "example-restore-registry-quay-database" as user "postgres".

  9. Create a pg_trgm extension of your Quay database by running the following command: 

    example_restore_registry_quay_database=> CREATE EXTENSION IF NOT EXISTS pg_trgm;

    Example output:

    CREATE EXTENSION

  10. Exit the postgres CLI by entering the following command: 

    \q

  11. Import the database backup to your new database by running the following command: 

    $ psql "host=172.24.10.50 port=5432 dbname=example_restore_registry_quay_database user=<username> password=test123"  -W <  /tmp/quay-backup/quay-backup.sql

    Example output:

    SET
SET
SET
SET
SET

    Update the value of DB_URI in your config.yaml from postgresql://<username>:test123@172.24.10.50/quay to postgresql://<username>:test123@172.24.10.50/example-restore-registry-quay-database before restarting the Red Hat Quay deployment.

    Note

    The DB_URI format is DB_URI postgresql://<login_user_name>:<login_user_password>@<postgresql_host>/<quay_database>. If you are moving from one PostgreSQL server to another PostgreSQL server, update the value of <login_user_name>, <login_user_password> and <postgresql_host> at the same time.

  12. In the /opt/new-quay-install directory, print the contents of your DISTRIBUTED_STORAGE_CONFIG bundle: 

    $ cat config.yaml | grep DISTRIBUTED_STORAGE_CONFIG -A10

    Example output:

    DISTRIBUTED_STORAGE_CONFIG:
   default:
DISTRIBUTED_STORAGE_CONFIG:
   default:
    - S3Storage
    - s3_bucket: <bucket_name>
      storage_path: /registry
      s3_access_key: <s3_access_key>
      s3_region: <region>
      s3_secret_key: <s3_secret_key>
      host: <host_name>

    Note

    Your DISTRIBUTED_STORAGE_CONFIG in /opt/new-quay-install must be updated before restarting your Red Hat Quay deployment.

  13. Export the AWS_ACCESS_KEY_ID by using the access_key credential obtained in Step 13: 

    $ export AWS_ACCESS_KEY_ID=<access_key>

  14. Export the AWS_SECRET_ACCESS_KEY by using the secret_key obtained in Step 13: 

    $ export AWS_SECRET_ACCESS_KEY=<secret_key>

  15. Create a new s3 bucket by entering the following command: 

    $ aws s3 mb s3://<new_bucket_name>  --region us-east-2

    Example output:

    $ make_bucket: quay

  16. Upload all blobs to the new s3 bucket by entering the following command: 

    $ aws s3 sync --no-verify-ssl \
--endpoint-url <example_endpoint_url>
/tmp/quay-backup/blob-backup/. s3://quay/

    where:

    <example_endpoint_url>

    Specifies the Red Hat Quay registry endpoint. The endpoint must be the same before backup and after restore.

    Example output:

    upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d to s3://quay/datastorage/registry/sha256/50/505edb46ea5d32b5cbe275eb766d960842a52ee77ac225e4dc8abb12f409a30d
upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0 to s3://quay/datastorage/registry/sha256/27/27930dc06c2ee27ac6f543ba0e93640dd21eea458eac47355e8e5989dea087d0
upload: ../../tmp/quay-backup/blob-backup/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec to s3://quay/datastorage/registry/sha256/8c/8c7daf5e20eee45ffe4b36761c4bb6729fb3ee60d4f588f712989939323110ec
...

  17. Before restarting your Red Hat Quay deployment, update the storage settings in your config.yaml file: 

    DISTRIBUTED_STORAGE_CONFIG:
   default:
DISTRIBUTED_STORAGE_CONFIG:
   default:
    - S3Storage
    - s3_bucket: <new_bucket_name>
      storage_path: /registry
      s3_access_key: <s3_access_key>
      s3_secret_key: <s3_secret_key>
      s3_region: <region>
      host: <host_name>
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top