1. Documentation
    2. history bug_report picture_as_pdf
×

Mirroring images for a disconnected installation by using the oc-mirror plugin v2

You can run your cluster in a restricted network without direct internet connectivity if you install the cluster from a mirrored set of OpenShift Container Platform container images in a private registry. This registry must be running whenever your cluster is running.

Just as you can use the oc-mirror OpenShift CLI (oc) plugin, you can also use oc-mirror plugin v2 to mirror images to a mirror registry in your fully or partially disconnected environments. To download the required images from the official Red Hat registries, you must run oc-mirror plugin v2 from a system with internet connectivity.

oc-mirror plugin v2 is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Prerequisites

  • If you do not have an existing solution for a container image registry, OpenShift Container Platform subscribers receive a mirror registry for Red Hat OpenShift. This mirror registry is included with your subscription and serves as a small-scale container registry. You can use this registry to mirror the necessary container images of OpenShift Container Platform for disconnected installations.

  • Every machine in the provisioned clusters must have access to the mirror registry. If the registry is unreachable, tasks like installation, updating, or routine operations such as workload relocation, might fail. Mirror registries must be operated in a highly available manner, ensuring their availability aligns with the production availability of your OpenShift Container Platform clusters.

High level workflow

The following steps outline the high-level workflow on how to mirror images to a mirror registry by using the oc-mirror plugin v2:

  1. Create an image set configuration file.

  2. Mirror the image set to the target mirror registry by using one of the following workflows:

    • Mirror an image set directly to the target mirror registry (mirror to mirror).

      • Mirror an image set to disk (Mirror-to-Disk), transfer the tar file to the target environment, then mirror the image set to the target mirror registry (Disk-to-Mirror).

  3. Configure your cluster to use the resources generated by the oc-mirror plugin v2.

  4. Repeat these steps to update your target mirror registry as necessary.

About oc-mirror plugin v2

The oc-mirror OpenShift CLI (oc) plugin is a single tool that mirrors all required OpenShift Container Platform content and other images to your mirror registry.

To use the new Technology Preview version of oc-mirror, add the --v2 flag to the oc-mirror plugin v2 command line.

oc-mirror plugin v2 has the following features:

  • Verifies that the complete image set specified in the image set config is mirrored to the mirrored registry, regardless of whether the images were previously mirrored or not.

  • Uses a cache system instead of metadata.

  • Maintains minimal archive sizes by incorporating only new images into the archive.

  • Generates mirroring archives with content selected by mirroring date.

  • Can generate ImageDigestMirrorSet (IDMS), ImageTagMirrorSet (ITMS), instead of ImageContentSourcePolicy (ICSP) for the full image set, rather than just for the incremental changes.

  • Saves filter Operator versions by bundle name.

  • Does not perform automatic pruning. V2 now has a Delete feature, which grants users more control over deleting images.

  • Introduces support for registries.conf. This change facilitates mirroring to multiple enclaves while using the same cache.

oc-mirror plugin v2 compatibility and support

The oc-mirror plugin v2 is supported for OpenShift Container Platform.

On aarch64, ppc64le, and s390x architectures the oc-mirror plugin v2 is supported only for OpenShift Container Platform versions 4.14 and later.

Use the latest available version of the oc-mirror plugin v2 regardless of which versions of OpenShift Container Platform you need to mirror.

Preparing your mirror hosts

To use the oc-mirror plugin v2 for image mirroring, you need to install the plugin and create a file with credentials for container images, enabling you to mirror from Red Hat to your mirror.

Installing the oc-mirror OpenShift CLI plugin

Install the oc-mirror OpenShift CLI plugin to manage image sets in disconnected environments.

Prerequisites

  • You have installed the OpenShift CLI (oc). If you are mirroring image sets in a fully disconnected environment, ensure the following:

    • You have installed the oc-mirror plugin on the host that has internet access.

    • The host in the disconnected environment has access to the target mirror registry.

  • You have set the umask parameter to 0022 on the operating system that uses oc-mirror.

  • You have installed the correct binary for the RHEL version that you are using.

Procedure

  1. Download the oc-mirror CLI plugin.

    1. Navigate to the Downloads page of the OpenShift Cluster Manager.

    2. Under the OpenShift disconnected installation tools section, click Download for OpenShift Client (oc) mirror plugin and save the file.

  2. Extract the archive:

    $ tar xvzf oc-mirror.tar.gz

  3. If necessary, update the plugin file to be executable:

    $ chmod +x oc-mirror

    Do not rename the oc-mirror file.

  4. Install the oc-mirror CLI plugin by placing the file in your PATH, for example, /usr/local/bin:

    $ sudo mv oc-mirror /usr/local/bin/.
Verification

  • Verify that the plugin for oc-mirror v2 is successfully installed by running the following command:

    $ oc mirror --v2 --help

Configuring credentials that allow images to be mirrored

Create a container image registry credentials file that enables you to mirror images from Red Hat to your mirror.

Do not use this image registry credentials file as the pull secret when you install a cluster. If you provide this file when you install cluster, all of the machines in the cluster will have write access to your mirror registry.

This process requires that you have write access to a container image registry on the mirror registry and adds the credentials to a registry pull secret.
Prerequisites

  • You configured a mirror registry to use in your disconnected environment.

  • You identified an image repository location on your mirror registry to mirror images into.

  • You provisioned a mirror registry account that allows images to be uploaded to that image repository.

Procedure

Complete the following steps on the installation host:

  1. Download your registry.redhat.io pull secret from Red Hat OpenShift Cluster Manager.

  2. Make a copy of your pull secret in JSON format:

    $ cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json> (1)
    1 Specify the path to the folder to store the pull secret in and a name for the JSON file that you create.

    The contents of the file resemble the following example:

    {
  "auths": {
    "cloud.openshift.com": {
      "auth": "b3BlbnNo...",
      "email": "you@example.com"
    },
    "quay.io": {
      "auth": "b3BlbnNo...",
      "email": "you@example.com"
    },
    "registry.connect.redhat.com": {
      "auth": "NTE3Njg5Nj...",
      "email": "you@example.com"
    },
    "registry.redhat.io": {
      "auth": "NTE3Njg5Nj...",
      "email": "you@example.com"
    }
  }
}

  3. Save the file as $XDG_RUNTIME_DIR/containers/auth.json.

  4. Generate the base64-encoded user name and password or token for your mirror registry:

    $ echo -n '<user_name>:<password>' | base64 -w0 (1)
BGVtbYk3ZHAtqXs=
    1 For <user_name> and <password>, specify the user name and password that you configured for your registry.

  5. Edit the JSON file and add a section that describes your registry to it:

      "auths": {
    "<mirror_registry>": { (1)
      "auth": "<credentials>", (2)
      "email": "you@example.com"
    }
  },
    1 Specify the registry domain name, and optionally the port, that your mirror registry uses to serve content. For example, registry.example.com or registry.example.com:8443
    2 Specify the base64-encoded user name and password for the mirror registry.

    The file resembles the following example:

    {
  "auths": {
    "registry.example.com": {
      "auth": "BGVtbYk3ZHAtqXs=",
      "email": "you@example.com"
    },
    "cloud.openshift.com": {
      "auth": "b3BlbnNo...",
      "email": "you@example.com"
    },
    "quay.io": {
      "auth": "b3BlbnNo...",
      "email": "you@example.com"
    },
    "registry.connect.redhat.com": {
      "auth": "NTE3Njg5Nj...",
      "email": "you@example.com"
    },
    "registry.redhat.io": {
      "auth": "NTE3Njg5Nj...",
      "email": "you@example.com"
    }
  }
}

Mirroring an image set to a mirror registry

Mirroring an image set to a mirror registry ensures that the required images are available in a secure and controlled environment, facilitating smoother deployments, updates, and maintenance tasks.

Building the image set configuration

The oc-mirror plugin v2 uses the image set configuration as the input file to determine the required images for mirroring.

Example for the ImageSetConfiguration input file
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  platform:
    channels:
    - name: stable-4.13
      minVersion: 4.13.10
      maxVersion: 4.13.10
    graph: true
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
      packages:
       - name: aws-load-balancer-operator
       - name: 3scale-operator
       - name: node-observability-operator
  additionalImages:
   - name: registry.redhat.io/ubi8/ubi:latest
   - name: registry.redhat.io/ubi9/ubi@sha256:20f695d2a91352d4eaa25107535126727b5945bff38ed36a3e59590f495046f0

Mirroring an image set in a partially disconnected environment

You can mirror image sets to a registry using the oc-mirror plugin v2 in environments with restricted internet access.

Prerequisites

  • You have access to the internet and the mirror registry in the environment where you are running the oc-mirror plugin v2.

Procedure

  • Mirror the images from the specified image set configuration to a specified registry by running the following command:

    $ oc mirror -c isc.yaml --workspace file://<file_path> docker://<mirror_registry_url> --v2 (1)
    1 Specify the URL or address of the mirror registry where the images are stored and from which they need to be deleted.
Verification

  1. Navigate to the cluster-resources directory within the working-dir directory that was generated in the <file_path> directory.

  2. Verify that the YAML files are present for the ImageDigestMirrorSet, ImageTagMirrorSet and CatalogSource resources.

Next steps

  • Configure your cluster to use the resources generated by oc-mirror plugin v2.

Mirroring an image set in a fully disconnected environment

You can mirror image sets in a fully disconnected environment where the OpenShift Container Platform cluster cannot access the public internet.

  1. Mirror to disk: Prepare an archive containing the image set for mirroring. Internet access is required.

  2. Manual step: Transfer the archive to the network of the disconnected mirror registry.

  3. Disk to mirror: To mirror the image set from the archive to the target disconnected registry, run oc-mirror plugin v2 from the environment that has access to the mirror registry.

Mirroring from mirror to disk

You can use the oc-mirror plugin v2 to generate an image set and save the content to disk. You can then transfer the generated image set can to the disconnected environment and mirrored to the target registry.

oc-mirror plugin v2 retrieves the container images from the source specified in the image set configuration and packs them into a tar archive in a local directory.

Procedure

  • Mirror the images from the specified image set configuration to the disk by running the following command:

    $ oc mirror -c isc.yaml file://<file_path> --v2 (1)
    1 Add the required file path.
Verification

  1. Navigate to the <file_path> directory that was generated.

  2. Verify that the archive files have been generated.

Next steps

  • Configure your cluster to use the resources generated by oc-mirror plugin v2.

Mirroring from disk to mirror

You can use the oc-mirror plugin v2 to mirror image sets from a disk to a target mirror registry.

The oc-mirror plugin v2 retrieves container images from a local disk and transfers them to the specified mirror registry.

Procedure

  • Process the image set file on the disk and mirror the contents to a target mirror registry by running the following command:

    $ oc mirror -c isc.yaml --from file://<file_path> docker://<mirror_registry_url> --v2 (1)
    1 Specify the URL or address of the mirror registry where the images are stored and from which they need to be deleted.
Verification

  1. Navigate to the cluster-resources directory within the working-dir directory that was generated in the <file_path> directory.

  2. Verify that the YAML files are present for the ImageDigestMirrorSet, ImageTagMirrorSet and CatalogSource resources.

Next steps

  • Configure your cluster to use the resources generated by oc-mirror plugin v2.

Additional resources

About custom resources generated by v2

With oc-mirror plugin v2, ImageDigestMirrorSet (IDMS) and ImageTagMirrorSet (ITMS) are generated by default if at least one image is found to which a tag refers. These sets contain mirrors for images referenced by digest or tag in releases, Operator catalogs and additional images.

The ImageDigestMirrorSet (IDMS) links the mirror registry to the source registry and forwards image pull requests using digest specifications. The ImagetagMirrorSet (ITMS) resource, however, redirects image pull requests by using image tags.

Operator Lifecycle Manager (OLM) uses the CatalogSource resource to retrieve information about the available Operators in the mirror registry.

The OSUS service uses the UpdateService resource to provide Cincinnati graph to the disconnected environment.

Configuring your cluster to use the resources generated by oc-mirror plugin v2

After you have mirrored your image set to the mirror registry, you must apply the generated ImageDigestMirrorSet (IDMS), ImageTagMirrorSet (ITMS), CatalogSource, and UpdateService to the cluster.

In oc-mirror plugin v2, the IDMS and ITMS files cover the entire image set, unlike the ICSP files in oc-mirror plugin v1. Therefore, the IDMS and ITMS files contain all images of the set even if you only add new images during incremental mirroring.
Prerequisites

  • You have access to the cluster as a user with the cluster-admin role.

Procedure

  • Apply the YAML files from the results directory to the cluster by running the following command:

    $ oc apply -f <path_to_oc-mirror_workspace>/working-dir/cluster-resources
Verification

  1. Verify that the ImageDigestMirrorSet resources are successfully installed by running the following command:

    $ oc get imagedigestmirrorset

  2. Verify that the ImageTagMirrorSet resources are successfully installed by running the following command:

    $ oc get imagetagmirrorset

  3. Verify that the CatalogSource resources are successfully installed by running the following command:

    $ oc get catalogsource -n openshift-marketplace

Deletion of images from your disconnected environment

Before you can use oc-mirror plugin v2, you must delete previously deployed images. oc-mirror plugin v2 no longer performs automatic pruning.

You must create the DeleteImageSetConfiguration file to delete image configuration when using oc-mirror plugin v2. This prevents accidentally deleting necessary or deployed images when making changes with ImageSetConfig.yaml.

In the following example, DeleteImageSetConfiguration removes the following:

  • All images of OpenShift Container Platform release 4.13.3.

  • The catalog image redhat-operator-index v4.12.

  • The aws-load-balancer-operator v0.0.1 bundle and all its related images.

  • The additional images ubi and ubi-minimal referenced by their corresponding digests.

Example: DeleteImageSetConfig
apiVersion: mirror.openshift.io/v2alpha1
kind: DeleteImageSetConfiguration
delete:
  platform:
    channels:
      - name: stable-4.13
        minVersion: 4.13.3
        maxVersion: 4.13.3
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.12
      packages:
      - name: aws-load-balancer-operator
         minVersion: 0.0.1
         maxVersion: 0.0.1
  additionalImages:
    - name: registry.redhat.io/ubi8/ubi@sha256:bce7e9f69fb7d4533447232478fd825811c760288f87a35699f9c8f030f2c1a6
    - name: registry.redhat.io/ubi8/ubi-minimal@sha256:8bedbe742f140108897fb3532068e8316900d9814f399d676ac78b46e740e34e

Consider using the mirror-to-disk and disk-to-mirror workflows to reduce mirroring issues.

In the image delete workflow, oc-mirror plugin v2 deletes only the manifests of the images, which does not reduce the storage occupied in the registry.

To free up storage space from unnecessary images, such as those with deleted manifests, you must enable the garbage collector on your container registry. With the garbage collector enabled, the registry will delete the image blobs that no longer have references to any manifests, thereby reducing the storage previously occupied by the deleted blobs. Enabling the garbage collector differs depending on your container registry.

To skip deleting the Operator catalog image when deleting images, you must list the specific Operators under the Operator catalog image in the DeleteImageSetConfiguration file. This ensures that only the specified Operators are deleted, not the catalog image.

If only the Operator catalog image is specified, all Operators within that catalog, as well as the catalog image itself, will be deleted.

Deleting the images from disconnected environment

To delete images from a disconnected environment using the oc-mirror plugin v2, follow the procedure.

Procedure

  1. Create a YAML file that deletes previous images:

    $ oc mirror delete --config delete-image-set-config.yaml --workspace file://<previously_mirrored_work_folder> --v2 --generate docker://<remote_registry>

    Where:

    • <previously_mirrored_work_folder>: Use the directory where images were previously mirrored or stored during the mirroring process.

    • <remote_registry>: Insert the URL or address of the remote container registry from which images will be deleted.

  2. Go to the <previously_mirrored_work_folder>/delete directory that was created.

  3. Verify that the delete-images.yaml file has been generated.

  4. Manually ensure that each image listed in the file is no longer needed by the cluster and can be safely removed from the registry.

  5. After you generate the delete YAML file, delete the images from the remote registry:

    $ oc mirror delete --v2 --delete-yaml-file <previously_mirrored_work_folder>/delete/delete-images.yaml docker:/ <remote_registry>

    Where:

    • <previously_mirrored_work_folder>: Specify your previously mirrored work folder.

      When using the mirror-to-mirror procedure, images are not cached locally, so you cannot delete images from a local cache.

Verifying your selected images for mirroring

You can use oc-mirror plugin v2 to perform a test run (dry run) that does not actually mirror any images. This enables you to review the list of images that would be mirrored. You can also use a dry run to catch any errors with your image set configuration early. When running a dry run on a mirror-to-disk workflow, the oc-mirror plugin v2 checks if all the images within the image set are available in its cache. Any missing images are listed in the missing.txt file. When a dry run is performed before mirroring, both missing.txt and mapping.txt files contain the same list of images.

Performing dry run for oc-mirror plugin v2

Verify your image set configuration by performing a dry run without mirroring any images. This ensures your setup is correct and prevents unintended changes.

Procedure

  • To perform a test run, run the oc mirror command and append the --dry-run argument to the command:

    $ oc mirror -c <image_set_config_yaml> --from file://<oc_mirror_workspace_path> docker://<mirror_registry_url> --dry-run --v2

    Where:

    • <image_set_config_yaml>: Use the image set configuration file that you just created.

    • <oc_mirror_workspace_path>: Insert the address of the workspace path.

    • <mirror_registry_url>: Insert the URL or address of the remote container registry from which images will be deleted.

      Example output
      $ oc mirror --config /tmp/isc_dryrun.yaml file://<oc_mirror_workspace_path> --dry-run --v2

[INFO]   : :warning:  --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready.
[INFO]   : :wave: Hello, welcome to oc-mirror
[INFO]   : :gear:  setting up the environment for you...
[INFO]   : :twisted_rightwards_arrows: workflow mode: mirrorToDisk
[INFO]   : :sleuth_or_spy:  going to discover the necessary images...
[INFO]   : :mag: collecting release images...
[INFO]   : :mag: collecting operator images...
[INFO]   : :mag: collecting additional images...
[WARN]   : :warning:  54/54 images necessary for mirroring are not available in the cache.
[WARN]   : List of missing images in : CLID-19/working-dir/dry-run/missing.txt.
please re-run the mirror to disk process
[INFO]   : :page_facing_up: list of all images for mirroring in : CLID-19/working-dir/dry-run/mapping.txt
[INFO]   : mirror time     : 9.641091076s
[INFO]   : :wave: Goodbye, thank you for using oc-mirror
Verification

  1. Navigate to the workspace directory that was generated:

    $ cd <oc_mirror_workspace_path>

  2. Review the mapping.txt and missing.txt files that were generated. These files contain a list of all images that would be mirrored.

Troubleshooting oc-mirror plugin v2 errors

oc-mirror plugin v2 now logs all image mirroring errors in a separate file, making it easier to track and diagnose failures.

When errors occur while mirroring release or release component images, they are critical. This stops the mirroring process immediately.

Errors with mirroring Operators, Operator-related images, or additional images do not stop the mirroring process. Mirroring continues, and oc-mirror plugin v2 logs updates every 8 images.

When an image fails to mirror, and that image is mirrored as part of one or more Operator bundles, oc-mirror plugin v2 notifies the user which Operators are incomplete, providing clarity on the Operator bundles affected by the error.

Procedure

  1. Check for server-related issues:

    Example error
    [ERROR]  : [Worker] error mirroring image localhost:55000/openshift/graph-image:latest error: copying image 1/4 from manifest list: trying to reuse blob sha256:edab65b863aead24e3ed77cea194b6562143049a9307cd48f86b542db9eecb6e at destination: pinging container registry localhost:5000: Get "https://localhost:5000/v2/": http: server gave HTTP response to HTTPS client

    1. Open the mirroring_error_date_time.log file in the working-dir/logs folder located in the oc-mirror plugin v2 output directory.

    2. Look for error messages that typically indicate server-side issues, such as HTTP 500 errors, expired tokens, or timeouts.

    3. Retry the mirroring process or contact support if the issue persists.

  2. Check for incomplete mirroring of Operators:

    Example error
    error mirroring image docker://registry.redhat.io/3scale-amp2/zync-rhel9@sha256:8bb6b31e108d67476cc62622f20ff8db34efae5d58014de9502336fcc479d86d (Operator bundles: [3scale-operator.v0.11.12] - Operators: [3scale-operator]) error: initializing source docker://localhost:55000/3scale-amp2/zync-rhel9:8bb6b31e108d67476cc62622f20ff8db34efae5d58014de9502336fcc479d86d: reading manifest 8bb6b31e108d67476cc62622f20ff8db34efae5d58014de9502336fcc479d86d in localhost:55000/3scale-amp2/zync-rhel9: manifest unknown
error mirroring image docker://registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:de0a70d1263a6a596d28bf376158056631afd0b6159865008a7263a8e9bf0c7d error: skipping operator bundle docker://registry.redhat.io/3scale-amp2/3scale-rhel7-operator-metadata@sha256:de0a70d1263a6a596d28bf376158056631afd0b6159865008a7263a8e9bf0c7d because one of its related images failed to mirror
error mirroring image docker://registry.redhat.io/3scale-amp2/system-rhel7@sha256:fe77272021867cc6b6d5d0c9bd06c99d4024ad53f1ab94ec0ab69d0fda74588e (Operator bundles: [3scale-operator.v0.11.12] - Operators: [3scale-operator]) error: initializing source docker://localhost:55000/3scale-amp2/system-rhel7:fe77272021867cc6b6d5d0c9bd06c99d4024ad53f1ab94ec0ab69d0fda74588e: reading manifest fe77272021867cc6b6d5d0c9bd06c99d4024ad53f1ab94ec0ab69d0fda74588e in localhost:55000/3scale-amp2/system-rhel7: manifest unknown

    1. Check for warnings in the console or log file indicating which Operators are incomplete.

      If an Operator is flagged as incomplete, the image related to that Operator likely failed to mirror.

    2. Manually mirror the missing image or retry the mirroring process.

  3. Check for errors related to generated cluster resources. Even if some images fail to mirror, oc-mirror v2 will still generate cluster resources such as IDMS.yaml and ITMS.yaml files for the successfully mirrored images.

    1. Check the output directory for the generated files.

    2. If these files are missing for specific images, ensure that no critical errors occurred for those images during the mirroring process.

By following these steps, you can better diagnose issues and ensure smoother mirroring.

Benefits of enclave support

Enclave support restricts internal access to a specific part of a network. Unlike a demilitarized zone (DMZ) network, which allows inbound and outbound traffic access through firewall boundaries, enclaves do not cross firewall boundaries.

Enclave Support is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The new enclave support functionality is for scenarios where mirroring is needed for multiple enclaves that are secured behind at least one intermediate disconnected network.

Enclave support has the following benefits:

  • You can mirror content for multiple enclaves and centralize it in a single internal registry. Because some customers want to run security checks on the mirrored content, with this setup they can run these checks all at once. The content is then vetted before being mirrored to downstream enclaves.

  • You can mirror content directly from the centralized internal registry to enclaves without restarting the mirroring process from the internet for each enclave.

  • You can minimize data transfer between network stages, so to ensure that a blob or image is transferred only once from one stage to another.

Enclave mirroring workflow

Enclave Support

The previous image outlines the flow for using the oc-mirror plugin in different environments, including environments with and without an internet connection.

Environment with Internet Connection:

  1. The user executes oc-mirror plugin v2 to mirror content from an online registry to a local disk directory.

  2. The mirrored content is saved to the disk for transfer to offline environments.

Disconnected Enterprise Environment (No Internet):

  • Flow 1:

    • The user runs oc-mirror plugin v2 to load the mirrored content from the disk directory, which was transferred from the online environment, into the enterprise-registry.in registry.

  • Flow 2:

    1. After updating the registries.conf file, the user executes the oc-mirror plugin v2 to mirror content from the enterprise-registry.in registry to an enclave environment.

    2. The content is saved to a disk directory for transfer to the enclave.

Enclave Environment (No Internet):

  • The user runs oc-mirror plugin v2 to load content from the disk directory into the enclave-registry.in registry.

The image visually represents the data flow across these environments and emphasizes the use of oc-mirror to handle disconnected and enclave environments without an internet connection.

Mirroring to an enclave

When you mirror to an enclave, you must first transfer the necessary images from one or more enclaves into the enterprise central registry.

The central registry is situated within a secure network, specifically a disconnected environment, and is not directly linked to the public internet. But the user must execute oc mirror in an environment with access to the public internet.

Procedure

  1. Before running oc-mirror plugin v2 in the disconnected environment, create a registries.conf file. The TOML format of the file is described in this specification:

    It is recommended to store the file under $HOME/.config/containers/registries.conf or /etc/containers/registries.conf.
    Example registries.conf
    [[registry]]
location="registry.redhat.io"
[[registry.mirror]]
location="<enterprise-registry.in>"

[[registry]]
location="quay.io"
[[registry.mirror]]
location="<enterprise-registry.in>"

  2. Generate a mirror archive.

    1. To collect all the OpenShift Container Platform content into an archive on the disk under <file_path>/enterprise-content, run the following command:

      $ oc mirror --v2 -c isc.yaml file://<file_path>/enterprise-content
      Example of isc.yaml
      apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
mirror:
  platform:
    architectures:
      - "amd64"
    channels:
      - name: stable-4.15
        minVersion: 4.15.0
        maxVersion: 4.15.3

      After the archive is generated, it is transferred to the disconnected environment. The transport mechanism is not part of oc-mirror plugin v2. The enterprise network administrators determine the transfer strategy.

      In some cases, the transfer is done manually, in that the disk is physically unplugged from one location, and plugged to another computer in the disconnected environment. In other cases, the Secure File Transfer Protocol (SFTP) or other protocols are used.

  3. After the transfer of the archive is done, you can execute oc-mirror plugin v2 again in order to mirror the relevant archive contents to the registry (entrerpise_registry.in in the example) as demonstrated in the following example:

    $ oc mirror --v2 -c isc.yaml --from file://<disconnected_environment_file_path>/enterprise-content docker://<enterprise_registry.in>/

    Where:

    • --from points to the folder containing the archive. It starts with the file://.

    • docker:// is the destination of the mirroring is the final argument. Because it is a docker registry.

    • -c (--config) is a mandatory argument. It enables oc-mirror plugin v2 to eventually mirror only sub-parts of the archive to the registry. One archive might contain several OpenShift Container Platform releases, but the disconnected environment or an enclave might mirror only a few.

  4. Prepare the imageSetConfig YAML file, which describes the content to mirror to the enclave:

    Example isc-enclave.yaml
    apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
mirror:
  platform:
    architectures:
      - "amd64"
    channels:
      - name: stable-4.15
        minVersion: 4.15.2
        maxVersion: 4.15.2

    You must run oc-mirror plugin v2 on a machine with access to the disconnected registry. In the previous example, the disconnected environment, enterprise-registry.in, is accessible.

  5. Update the graph URL

    If you are using graph:true, oc-mirror plugin v2 attempts to reach the cincinnati API endpoint. Because this environment is disconnected, be sure to export the environment variable UPDATE_URL_OVERRIDE to refer to the URL for the OpenShift Update Service (OSUS):

    $ export UPDATE_URL_OVERRIDE=https://<osus.enterprise.in>/graph

    For more information on setting up OSUS on an OpenShift cluster, see "Updating a cluster in a disconnected environment using the OpenShift Update Service".

When upgrading OpenShift Container Platform Extended Update Support (EUS) versions, an intermediate version might be required between the current and target versions. For example, if the current version is 4.14 and target version is 4.16, you might need to include a version such as 4.15.8 in the ImageSetConfiguration when using the oc-mirror plugin v2.

The oc-mirror plugin v2 might not always detect this automatically, so check the Cincinnati graph web page to confirm any required intermediate versions and add them manually to your configuration.

  1. Generate a mirror archive from the enterprise registry for the enclave.

    To prepare an archive for the enclave1, the user executes oc-mirror plugin v2 in the enterprise disconnected environment by using the imageSetConfiguration specific for that enclave. This ensures that only images needed by that enclave are mirrored:

    $ oc mirror --v2 -c isc-enclave.yaml
file:///disk-enc1/

    This action collects all the OpenShift Container Platform content into an archive and generates an archive on disk.

  2. After the archive is generated, it will be transferred to the enclave1 network. The transport mechanism is not the responsibility of oc-mirror plugin v2.

  3. Mirror contents to the enclave registry

    After the transfer of the archive is done, the user can execute oc-mirror plugin v2 again in order to mirror the relevant archive contents to the registry.

    $ oc mirror --v2 -c isc-enclave.yaml --from file://local-disk docker://registry.enc1.in

    The administrators of the OpenShift Container Platform cluster in enclave1 are now ready to install or upgrade that cluster.

How filtering works in the operator catalog

oc-mirror plugin v2 selects the list of bundles for mirroring by processing the information in imageSetConfig.

When oc-mirror plugin v2 selects bundles for mirroring, it does not infer Group Version Kind (GVK) or bundle dependencies, omitting them from the mirroring set. Instead, it strictly adheres to the user instructions. You must explicitly specify any required dependent packages and their versions.

Bundle versions typically use semantic versioning standards (SemVer), and you can sort bundles within a channel by version. You can select buncles that fall within a specific range in the ImageSetConfig.

This selection algorithm ensures consistent outcomes compared to oc-mirror plugin v1. However, it does not include upgrade graph details, such as replaces, skip, and skipRange. This approach differs from the OLM algorithm. It might mirror more bundles than necessary for upgrading a cluster because of potentially shorter upgrade paths between the minVersion and maxVersion.

Table 1. Use the following table to see what bundle versions are included in different scenarios
ImageSetConfig operator filtering Expected bundle versions

Scenario 1

mirror:
 operators:
   - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.10

For each package in the catalog, 1 bundle, corresponding to the head version of the default channel for that package.

Scenario 2

mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.10
      full: true

All bundles of all channels of the specified catalog

Scenario 3

mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.10
     packages:
    - name: compliance-operator

One bundle, corresponding to the head version of the default channel for that package

Scenario 4

mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.10
      full: true
      - packages:
          - name: elasticsearch-operator

All bundles of all channels for the packages specified

Scenario 5

mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
    packages:
    - name: compliance-operator
       minVersion: 5.6.0

All bundles in the default channel, from the minVersion, up to the channel head for that package that do not rely on the shortest path from upgrade the graph.

Scenario 6

mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
    packages:
    - name: compliance-operator
        maxVersion: 6.0.0

All bundles in the default channel that are lower than the maxVersion for that package.

Scenario 7

mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
    packages:
    - name: compliance-operator
        minVersion: 5.6.0
        maxVersion: 6.0.0

All bundles in the default channel, between the minVersion and maxVersion for that package. The head of the channel is not included, even if multiple channels are included in the filtering.

Scenario 8

mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
    packages:
    - name: compliance-operator
        channels
          - name: stable

The head bundle for the selected channel of that package.

Scenario 9

mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.10
      full: true
      - packages:
          - name: elasticsearch-operator
            channels:
               - name: 'stable-v0'

All bundles for the specified packages and channels.

Scenario 10

mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
    packages:
    - name: compliance-operator
        channels
          - name: stable
          - name: stable-5.5

The head bundle for each selected channel of that package.

Scenario 11

mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
    packages:
    - name: compliance-operator
        channels
          - name: stable
            minVersion: 5.6.0

Within the selected channel of that package, all versions starting with the minVersion up to the channel head. This scenario does not relyon the shortest path from the upgrade graph.

Scenario 12

mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
    packages:
    - name: compliance-operator
        channels
          - name: stable
            maxVersion: 6.0.0

Within the selected channel of that package, all versions up to the maxVersion (not relying on the shortest path from the upgrade graph). The head of the channel is not included, even if multiple channels are included in the filtering.

Scenario 13

mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
    packages:
    - name: compliance-operator
       channels
          - name: stable
            minVersion: 5.6.0
            maxVersion: 6.0.0

Within the selected channel of that package, all versions between the minVersion and maxVersion, not relying on the shortest path from the upgrade graph. The head of the channel is not included, even if multiple channels are included in the filtering.

Scenario 14

mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
    packages:
    - name: aws-load-balancer-operator
      bundles:
      - name: aws-load-balancer-operator.v1.1.0
    - name: 3scale-operator
      bundles:
      - name: 3scale-operator.v0.10.0-mas

Only the bundles specified for each package are included in the filtering.

Scenario 15

mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
    packages:
    - name: compliance-operator
        channels
          - name: stable
        minVersion: 5.6.0
        maxVersion: 6.0.0

Do not use this scenario. filtering by channel and by package with a minVersion or maxVersion is not allowed.

Scenario 16

mirror:
  operators:
   - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
    packages:
    - name: compliance-operator
        channels
          - name: stable
        minVersion: 5.6.0
        maxVersion: 6.0.0

Do not use this scenario. You cannot filter using full:true and the minVersion or maxVersion.

Scenario 17

mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.15
      full: true
    packages:
    - name: compliance-operator
        channels
          - name: stable
            minVersion: 5.6.0
            maxVersion: 6.0.0

Do not use this scenario. You cannot filter using full:true and the minVersion or maxVersion.

ImageSet configuration parameters for oc-mirror plugin v2

The oc-mirror plugin v2 requires an image set configuration file that defines what images to mirror. The following table lists the available parameters for the ImageSetConfiguration resource.

Using the minVersion and maxVersion properties to filter for a specific Operator version range can result in a multiple channel heads error. The error message states that there are multiple channel heads. This is because when the filter is applied, the update graph of the Operator is truncated.

OLM requires that every Operator channel contains versions that form an update graph with exactly one end point, that is, the latest version of the Operator. When the filter range is applied, that graph can turn into two or more separate graphs or a graph that has more than one end point.

To avoid this error, do not filter out the latest version of an Operator. If you still run into the error, depending on the Operator, either the maxVersion property must be increased or the minVersion property must be decreased. Because every Operator graph can be different, you might need to adjust these values until the error resolves.
Table 2. ImageSetConfiguration parameters
Parameter Description Values

apiVersion

The API version of the ImageSetConfiguration content.

String Example: mirror.openshift.io/v2alpha1

archiveSize

The maximum size, in GiB, of each archive file within the image set.

Integer Example: 4

mirror

The configuration of the image set.

Object

mirror.additionalImages

The additional images configuration of the image set.

Array of objects

Example:

additionalImages:
  - name: registry.redhat.io/ubi8/ubi:latest

mirror.additionalImages.name

The tag or digest of the image to mirror.

String Example: registry.redhat.io/ubi8/ubi:latest

mirror.blockedImages

The full tag, digest, or pattern of images to block from mirroring.

Array of strings Example: docker.io/library/alpine

mirror.operators

The Operators configuration of the image set.

Array of objects

Example:

operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:4.16
    packages:
      - name: elasticsearch-operator
        minVersion: '2.4.0'

mirror.operators.catalog

The Operator catalog to include in the image set.

String Example: registry.redhat.io/redhat/redhat-operator-index:v4.15

mirror.operators.full

When true, downloads the full catalog, Operator package, or Operator channel.

Boolean The default value is false.

mirror.operators.packages

The Operator packages configuration.

Array of objects

Example:

operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:4.16
    packages:
      - name: elasticsearch-operator
        minVersion: '5.2.3-31'

mirror.operators.packages.name

The Operator package name to include in the image set.

String Example: elasticsearch-operator

mirror.operators.packages.channels

Operator package channel configuration

Object

mirror.operators.packages.channels.name

The Operator channel name, unique within a package, to include in the image set.

String Eample: fast or stable-v4.15

mirror.operators.packages.channels.maxVersion

The highest version of the Operator mirror across all channels in which it exists.

String Example: 5.2.3-31

mirror.operators.packages.channels.minVersion

The lowest version of the Operator to mirror across all channels in which it exists

String Example: 5.2.3-31

mirror.operators.packages.maxVersion

The highest version of the Operator to mirror across all channels in which it exists.

String Example: 5.2.3-31

mirror.operators.packages.minVersion

The lowest version of the Operator to mirror across all channels in which it exists.

String Example: 5.2.3-31

mirror.operators.packages.bundles

Selected bundles configuration

Array of objects

Example:

operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:4.16
    packages:
    - name: 3scale-operator
      bundles:
      - name: 3scale-operator.v0.10.0-mas

mirror.operators.packages.bundles.name

Name of the bundle selected for mirror (as it appears in the catalog).

String Example : 3scale-operator.v0.10.0-mas

mirror.operators.targetCatalog

An alternative name and optional namespace hierarchy to mirror the referenced catalog as

String Example: my-namespace/my-operator-catalog

mirror.operators.targetCatalogSourceTemplate

Path on disk for a template to use to complete catalogSource custom resource generated by oc-mirror plugin v2.

String Example: /tmp/catalog-source_template.yaml Example of a template file:

apiVersion: operators.coreos.com/v2alpha1
kind: CatalogSource
metadata:
  name: discarded
  namespace: openshift-marketplace
spec:
  image: discarded
  sourceType: grpc
  updateStrategy:
registryPoll:
interval: 30m0s

mirror.operators.targetTag

An alternative tag to append to the targetName or targetCatalog.

String Example: v1

mirror.platform

The platform configuration of the image set.

Object

mirror.platform.architectures

The architecture of the platform release payload to mirror.

Array of strings Example:

architectures:
  - amd64
  - arm64
  - multi
  - ppc64le
  - s390x

The default value is amd64. The value multi ensures that the mirroring is supported for all available architectures, eliminating the need to specify individual architectures

mirror.platform.channels

The platform channel configuration of the image set.

Array of objects Example:

channels:
  - name: stable-4.12
  - name: stable-4.16

mirror.platform.channels.full

When true, sets the minVersion to the first release in the channel and the maxVersion to the last release in the channel.

Boolean The default value is false

mirror.platform.channels.name

Name of the release channel

String Example: stable-4.15

mirror.platform.channels.minVersion

The minimum version of the referenced platform to be mirrored.

String Example: 4.12.6

mirror.platform.channels.maxVersion

The highest version of the referenced platform to be mirrored.

String Example: 4.15.1

mirror.platform.channels.shortestPath

Toggles shortest path mirroring or full range mirroring.

Boolean The default value is false

mirror.platform.channels.type

Type of the platform to be mirrored

String Example: ocp or okd. The default is ocp.

mirror.platform.graph

Indicates whether the OSUS graph is added to the image set and subsequently published to the mirror.

Boolean The default value is false

Delete ImageSet Configuration parameters

To use the oc-mirror plugin v2, you must have delete image set configuration file that defines which images to delete from the mirror registry. The following table lists the available parameters for the DeleteImageSetConfiguration resource.

Table 3. DeleteImageSetConfiguration parameters
Parameter Description Values

apiVersion

The API version for the DeleteImageSetConfiguration content.

String Example: mirror.openshift.io/v2alpha1

delete

The configuration of the image set to delete.

Object

delete.additionalImages

The additional images configuration of the delete image set.

Array of objects Example:

additionalImages:
  - name: registry.redhat.io/ubi8/ubi:latest

delete.additionalImages.name

The tag or digest of the image to delete.

String Example: registry.redhat.io/ubi8/ubi:latest

delete.operators

The Operators configuration of the delete image set.

Array of objects Example:

operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:{product-version}
    packages:
      - name: elasticsearch-operator
        minVersion: '2.4.0'

delete.operators.catalog

The Operator catalog to include in the delete image set.

String Example: registry.redhat.io/redhat/redhat-operator-index:v4.15

delete.operators.full

When true, deletes the full catalog, Operator package, or Operator channel.

Boolean The default value is false

delete.operators.packages

Operator packages configuration

Array of objects Example:

operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:{product-version}
    packages:
      - name: elasticsearch-operator
        minVersion: '5.2.3-31'

delete.operators.packages.name

The Operator package name to include in the delete image set.

String Example: elasticsearch-operator

delete.operators.packages.channels

Operator package channel configuration

Object

delete.operators.packages.channels.name

The Operator channel name, unique within a package, to include in the delete image set.

String Example: fast or stable-v4.15

delete.operators.packages.channels.maxVersion

The highest version of the Operator to delete within the selected channel.

String Example: 5.2.3-31

delete.operators.packages.channels.minVersion

The lowest version of the Operator to delete within the selection in which it exists.

String Example: 5.2.3-31

delete.operators.packages.maxVersion

The highest version of the Operator to delete across all channels in which it exists.

String Example: 5.2.3-31

delete.operators.packages.minVersion

The lowest version of the Operator to delete across all channels in which it exists.

String Example: 5.2.3-31

delete.operators.packages.bundles

The selected bundles configuration

Array of objects

You cannot choose both channels and bundles for the same operator.

Example:

operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:{product-version}
    packages:
    - name: 3scale-operator
      bundles:
      - name: 3scale-operator.v0.10.0-mas

delete.operators.packages.bundles.name

Name of the bundle selected to delete (as it is displayed in the catalog)

String Example : 3scale-operator.v0.10.0-mas

delete.platform

The platform configuration of the image set

Object

delete.platform.architectures

The architecture of the platform release payload to delete.

Array of strings Example:

architectures:
  - amd64
  - arm64
  - multi
  - ppc64le
  - s390x

The default value is amd64

delete.platform.channels

The platform channel configuration of the image set.

Array of objects

Example:

channels:
  - name: stable-4.12
  - name: stable-4.16

delete.platform.channels.full

When true, sets the minVersion to the first release in the channel and the maxVersion to the last release in the channel.

Boolean The default value is false

delete.platform.channels.name

Name of the release channel

String Example: stable-4.15

delete.platform.channels.minVersion

The minimum version of the referenced platform to be deleted.

String Example: 4.12.6

delete.platform.channels.maxVersion

The highest version of the referenced platform to be deleted.

String Example: 4.15.1

delete.platform.channels.shortestPath

Toggles between deleting the shortest path and deleting the full range.

Boolean The default value is false

delete.platform.channels.type

Type of the platform to be deleted

String Example: ocp or okd The default is ocp

delete.platform.graph

Determines whether the OSUS graph is deleted as well on the mirror registry as well.

Boolean The default value is false

Command reference for oc-mirror plugin v2

The following tables describe the oc mirror subcommands and flags for oc-mirror plugin v2:

Table 4. Subcommands and flags for the oc-mirror plugin v2
Subcommand Description

help

Show help about any subcommand

version

Output the oc-mirror version

delete

Deletes images in remote registry and local cache.
Table 5. oc mirror flags
Flag Description

--authfile

Displays the string path of the authentication file. Default is ${XDG_RUNTIME_DIR}/containers/auth.json.

-c, --config <string>

Specifies the path to an image set configuration file.

--dest-tls-verify

Requires HTTPS and verifies certificates when accessing the container registry or daemon.

--dry-run

Prints actions without mirroring images

--from <string>

Specifies the path to an image set archive that was generated by executing oc-mirror plugin v2 to load a target registry.

-h, --help

Displays help

--loglevel

Displays string log levels. Supported values include info, debug, trace, error. The default is info.

-p, --port

Determines the HTTP port used by oc-mirror plugin v2 local storage instance. The default is 55000.

--max-nested-paths <int>

Specifies the maximum number of nested paths for destination registries that limit nested paths. The default is 0.

--secure-policy

Default value is false. If you set a non-default value, the command enables signature verification, which is the secure policy for signature verification.

--since

Includes all new content since a specified date (format: yyyy-mm-dd). When not provided, new content since previous mirroring is mirrored.

--src-tls-verify

Requires HTTPS and verifies certificates when accessing the container registry or daemon.

--strict-archive

Default value is false. If you set a value, the command generates archives that are strictly less than the archiveSize that was set in the imageSetConfig custom resource (CR). Mirroring exist in error if a file being archived exceeds archiveSize (GB).

-v, --version

Displays the version for oc-mirror plugin v2.

--workspace

Determines string oc-mirror plugin v2 workspace where resources and internal artifacts are generated.