Chapter 15. Geo-replication

Procedure

1. Update your config.yaml file to include the storage engines to which data is replicated. You must list all storage engines to be used:

   # ...
   FEATURE_STORAGE_REPLICATION: true
   # ...
   DISTRIBUTED_STORAGE_CONFIG:
       usstorage:
           - RHOCSStorage
           - access_key: <access_key>
             bucket_name: <example_bucket>
             hostname: my.noobaa.hostname
             is_secure: false
             port: "443"
             secret_key: <secret_key>
             storage_path: /datastorage/registry
       eustorage:
           - S3Storage
           - host: s3.amazon.com
             port: "443"
             s3_access_key: <access_key>
             s3_bucket: <example bucket>
             s3_secret_key: <secret_key>
             storage_path: /datastorage/registry
   DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
   DISTRIBUTED_STORAGE_PREFERENCE:
       - usstorage
       - eustorage
   # ...

2. Optional. If complete replication of all images to all storage engines is required, you can replicate images to the storage engine by manually setting the DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS field. This ensures that all images are replicated to that storage engine. For example:

   # ...
   DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
       - usstorage
       - eustorage
   # ...

   Note

   To enable per-namespace replication, contact Red Hat Quay support.

3. After adding storage and enabling Replicate to storage engine by default for geo-replication, you must sync existing image data across all storage. To do this, you must execute into the container by running the following command:

   $ podman exec -it <container_id>

4. To sync the content after adding new storage, enter the following commands:

   # scl enable python27 bash

   # python -m util.backfillreplication

   Note

   This is a one time operation to sync content after adding new storage.

Procedure

1. Copy the config.yaml file to all machines running Red Hat Quay.

2. For each machine in each region, add a QUAY_DISTRIBUTED_STORAGE_PREFERENCE environment variable with the preferred storage engine for the region in which the machine is running.

   For example, for a machine running in Europe with the config directory on the host available from $QUAY/config:

   $ sudo podman run -d --rm -p 80:8080 -p 443:8443  \
      --name=quay \
      -v $QUAY/config:/conf/stack:Z \
      -e QUAY_DISTRIBUTED_STORAGE_PREFERENCE=europestorage \
      registry.redhat.io/quay/quay-rhel9:v3.18.0

   Note

   The value of the environment variable specified must match the name of a Location ID as defined in the config panel.

3. Restart all Red Hat Quay containers.

Procedure

1. Sync the blobs between all of your defined sites by running the following command:

   $ python -m util.backfillreplication

   Warning

   Prior to removing storage engines from your Red Hat Quay config.yaml file, you must ensure that all blobs are synced between all defined sites. Complete this step before proceeding.

2. In your Red Hat Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

3. Enter the following command to obtain a list of running containers:

   $ podman ps

   Example output:

   CONTAINER ID  IMAGE                                                                     COMMAND         CREATED         STATUS             PORTS                                        NAMES
   92c5321cde38  registry.redhat.io/rhel8/redis-5:1                                        run-redis       11 days ago     Up 11 days ago     0.0.0.0:6379->6379/tcp                       redis
   4e6d1ecd3811  registry.redhat.io/rhel8/postgresql-13:1-109                              run-postgresql  33 seconds ago  Up 34 seconds ago  0.0.0.0:5432->5432/tcp                       postgresql-quay
   d2eadac74fda  registry-proxy.engineering.redhat.com/rh-osbs/quay-quay-rhel8:v3.9.0-131  registry        4 seconds ago   Up 4 seconds ago   0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp  quay

4. Enter the following command to execute a shell inside of the PostgreSQL container:

   $ podman exec -it postgresql-quay -- /bin/bash

5. Enter psql by running the following command:

   bash-4.4$ psql

6. Enter the following command to reveal a list of sites in your geo-replicated deployment:

   quay=# select * from imagestoragelocation;

   Example output:

    id |       name
   ----+-------------------
     1 | usstorage
     2 | eustorage

7. Enter the following command to exit the postgres CLI to re-enter bash-4.4:

   \q

8. Enter the following command to permanently remove the eustorage site:

   Important

   The following action cannot be undone. Use with caution.

   bash-4.4$ python -m util.removelocation eustorage

   Example output:

   WARNING: This is a destructive operation. Are you sure you want to remove eustorage from your storage locations? [y/n] y
   Deleted placement 30
   Deleted placement 31
   Deleted placement 32
   Deleted placement 33
   Deleted location eustorage

Procedure

1. Deploy a PostgreSQL instance for Red Hat Quay.

2. Log in to the database by entering the following command:

   psql -U <username> -h <hostname> -p <port> -d <database_name>

3. Create a database for Red Hat Quay named quay. For example:

   CREATE DATABASE quay;

4. Enable the pg_trgm extension inside the database:

   \c quay;
   CREATE EXTENSION IF NOT EXISTS pg_trgm;

5. Deploy a Redis instance:

   Note
   • Deploying a Redis instance might be unnecessary if your cloud provider has its own service.
   • Deploying a Redis instance is required if you are leveraging Builders.
   1. Deploy a VM for Redis.
   2. Verify that the Redis instance is accessible from the clusters where Red Hat Quay is running.
   3. Port 6379/TCP must be open.

   4. Run Redis inside the instance:

      sudo dnf install -y podman
      podman run -d --name redis -p 6379:6379 redis

6. Create two object storage backends, one for each cluster. Ideally, one object storage bucket is close to the first, or primary, cluster, and the other runs closer to the second, or secondary, cluster.
7. Deploy the clusters with the same config bundle, using environment variable overrides to select the appropriate storage backend for an individual cluster.
8. Configure a load balancer to provide a single entry point to the clusters.

Procedure

1. Sync the blobs between all of your defined sites by running the following command:

   $ python -m util.backfillreplication

   Warning

   Prior to removing storage engines from your Red Hat Quay config.yaml file, you must ensure that all blobs are synced between all defined sites.

   When running this command, replication jobs are created which are picked up by the replication worker. If blobs need to be replicated, the script returns UUIDs of blobs that are replicated. If you run this command multiple times, and the output from the return script is empty, it does not mean that the replication process is done; it means that no more blobs remain to be queued for replication. Customers should use appropriate judgement before proceeding, as the allotted time replication takes depends on the number of blobs detected.

   Alternatively, you could use a third party cloud tool, such as Microsoft Azure, to check the synchronization status.

   This step must be completed before proceeding.

2. In your Red Hat Quay config.yaml file for site usstorage, remove the DISTRIBUTED_STORAGE_CONFIG entry for the eustorage site.

3. Identify your Red Hat Quay application pods by entering the following command:

   $ oc get pod -n <quay_namespace>

   Example output:

   quay390usstorage-quay-app-5779ddc886-2drh2
   quay390eustorage-quay-app-66969cd859-n2ssm

4. Open an interactive shell session in the usstorage pod by entering the following command:

   $ oc rsh quay390usstorage-quay-app-5779ddc886-2drh2

5. Permanently remove the eustorage site by entering the following command:

   Important

   The following action cannot be undone. Use with caution.

   sh-4.4$ python -m util.removelocation eustorage

   Example output:

   WARNING: This is a destructive operation. Are you sure you want to remove eustorage from your storage locations? [y/n] y
   Deleted placement 30
   Deleted placement 31
   Deleted placement 32
   Deleted placement 33
   Deleted location eustorage