When OpenShift Container Platform is installed on restricted networks, also known as a disconnected cluster, Operator Lifecycle Manager (OLM) can no longer use the default OperatorHub sources because they require full Internet connectivity. Cluster administrators can disable those default sources and create local mirrors so that OLM can install and manage Operators from the local sources instead.

While OLM can manage Operators from local sources, the ability for a given Operator to run successfully in a restricted network still depends on the Operator itself. The Operator must:

  • List any related images, or other container images that the Operator might require to perform their functions, in the relatedImages parameter of its ClusterServiceVersion (CSV) object.

  • Reference all specified images by a digest (SHA) and not by a tag.

See the following Red Hat Knowledgebase Article for a list of Red Hat Operators that support running in disconnected mode:

Understanding Operator catalog images

Operator Lifecycle Manager (OLM) always installs Operators from the latest version of an Operator catalog. As of OpenShift Container Platform 4.3, Red Hat-provided Operators are distributed via Quay App Registry catalogs from quay.io.

As catalogs are updated, the latest versions of Operators change, and older versions may be removed or altered. This behavior can cause problems maintaining reproducible installs over time. In addition, when OLM runs on an OpenShift Container Platform cluster in a restricted network environment, it is unable to access the catalogs from quay.io directly.

Using the oc adm catalog build command, cluster administrators can create an Operator catalog image. An Operator catalog image is:

  • a point-in-time export of an App Registry type catalog’s content.

  • the result of converting an App Registry catalog to a container image type catalog.

  • an immutable artifact.

Creating an Operator catalog image provides a simple way to use this content without incurring the aforementioned issues.

Building an Operator catalog image

Cluster administrators can build a custom Operator catalog image to be used by Operator Lifecycle Manager (OLM) and push the image to a container image registry that supports Docker v2-2. For a cluster on a restricted network, this registry can be a registry that the cluster has network access to, such as the mirror registry created during the restricted network installation.

The OpenShift Container Platform cluster’s internal registry cannot be used as the target registry because it does not support pushing without a tag, which is required during the mirroring process.

For this example, the procedure assumes use of the mirror registry that has access to both your network and the internet.

Prerequisites
  • A Linux workstation with unrestricted network access [1]

  • oc version 4.3.5+

  • podman version 1.4.4+

  • Access to mirror registry that supports Docker v2-2

  • If you are working with private registries, set the REG_CREDS environment variable to the file path of your registry credentials for use in later steps. For example, for the podman CLI:

    $ REG_CREDS=${XDG_RUNTIME_DIR}/containers/auth.json
Procedure
  1. On the workstation with unrestricted network access, authenticate with the target mirror registry:

    $ podman login <registry_host_name>

    Also authenticate with registry.redhat.io so that the base image can be pulled during the build:

    $ podman login registry.redhat.io
  2. Build a catalog image based on the redhat-operators catalog from quay.io, tagging and pushing it to your mirror registry:

    $ oc adm catalog build \
        --appregistry-org redhat-operators \(1)
        --from=registry.redhat.io/openshift4/ose-operator-registry:v4.4 \(2)
        --filter-by-os="linux/amd64" \(3)
        --to=<registry_host_name>:<port>/olm/redhat-operators:v1 \(4)
        [-a ${REG_CREDS}] \(5)
        [--insecure] (6)
    
    INFO[0013] loading Bundles                               dir=/var/folders/st/9cskxqs53ll3wdn434vw4cd80000gn/T/300666084/manifests-829192605
    ...
    Pushed sha256:f73d42950021f9240389f99ddc5b0c7f1b533c054ba344654ff1edaf6bf827e3 to example_registry:5000/olm/redhat-operators:v1
    1 Organization (namespace) to pull from an App Registry instance.
    2 Set --from to the ose-operator-registry base image using the tag that matches the target OpenShift Container Platform cluster major and minor version.
    3 Set --filter-by-os to the operating system and architecture to use for the base image, which must match the target OpenShift Container Platform cluster. Valid values are linux/amd64, linux/ppc64le, and linux/s390x.
    4 Name your catalog image and include a tag, for example, v1.
    5 Optional: If required, specify the location of your registry credentials file.
    6 Optional: If you do not want to configure trust for the target registry, add the --insecure flag.

    Sometimes invalid manifests are accidentally introduced into Red Hat’s catalogs; when this happens, you might see some errors:

    ...
    INFO[0014] directory                                     dir=/var/folders/st/9cskxqs53ll3wdn434vw4cd80000gn/T/300666084/manifests-829192605 file=4.2 load=package
    W1114 19:42:37.876180   34665 builder.go:141] error building database: error loading package into db: fuse-camel-k-operator.v7.5.0 specifies replacement that couldn't be found
    Uploading ... 244.9kB/s

    These errors are usually non-fatal, and if the Operator package mentioned does not contain an Operator you plan to install or a dependency of one, then they can be ignored.

Configuring OperatorHub for restricted networks

Cluster administrators can configure OLM and OperatorHub to use local content in a restricted network environment using a custom Operator catalog image. For this example, the procedure uses a custom redhat-operators catalog image previously built and pushed to a supported registry.

Prerequisites
  • A Linux workstation with unrestricted network access [1]

  • A custom Operator catalog image pushed to a supported registry

  • oc version 4.3.5+

  • podman version 1.4.4+

  • Access to mirror registry that supports Docker v2-2

  • If you are working with private registries, set the REG_CREDS environment variable to the file path of your registry credentials for use in later steps. For example, for the podman CLI:

    $ REG_CREDS=${XDG_RUNTIME_DIR}/containers/auth.json
Procedure
  1. Disable the default OperatorSources by adding disableAllDefaultSources: true to the spec:

    $ oc patch OperatorHub cluster --type json \
        -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

    This disables the default OperatorSources that are configured by default during an OpenShift Container Platform installation.

  2. Mirror the contents of your catalog to your target registry. The following oc adm catalog mirror command extracts the contents of your custom Operator catalog image to generate the manifests required for mirroring and mirrors the images to your registry:

    $ oc adm catalog mirror \
        <registry_host_name>:<port>/olm/redhat-operators:v1 \(1)
        <registry_host_name>:<port> \
        [-a ${REG_CREDS}] \(2)
        [--insecure] \(3)
        [--filter-by-os="<os>/<arch>"] (4)
    
    mirroring ...
    1 Specify your Operator catalog image.
    2 Optional: If required, specify the location of your registry credentials file.
    3 Optional: If you do not want to configure trust for the target registry, add the --insecure flag.
    4 Optional: Because the catalog might reference images that support multiple architectures and operating systems, you can filter by architecture and operating system to mirror only the images that match. Valid values are linux/amd64, linux/ppc64le, and linux/s390x.

    This command also creates a <image_name>-manifests/ directory in the current directory and generates the following files:

    • The imageContentSourcePolicy.yaml file defines an ImageContentSourcePolicy object that can configure nodes to translate between the image references stored in Operator manifests and the mirrored registry.

    • The mapping.txt file contains all of the source images and where to map them in the target registry. This file is compatible with the oc image mirror command and can be used to further customize the mirroring configuration.

  3. Apply the manifests:

    $ oc apply -f ./redhat-operators-manifests
  4. Create a CatalogSource object that references your catalog image.

    1. Modify the following to your specifications and save it as a catalogsource.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        name: my-operator-catalog
        namespace: openshift-marketplace
      spec:
        sourceType: grpc
        image: <registry_host_name>:<port>/olm/redhat-operators:v1 (1)
        displayName: My Operator Catalog
        publisher: grpc
      1 Specify your custom Operator catalog image.
    2. Use the file to create the CatalogSource object:

      $ oc create -f catalogsource.yaml
  5. Verify the CatalogSource and package manifest are created successfully:

    # oc get pods -n openshift-marketplace
    NAME READY STATUS RESTARTS AGE
    my-operator-catalog-6njx6 1/1 Running 0 28s
    marketplace-operator-d9f549946-96sgr 1/1 Running 0 26h
    
    # oc get catalogsource -n openshift-marketplace
    NAME DISPLAY TYPE PUBLISHER AGE
    my-operator-catalog My Operator Catalog grpc 5s
    
    # oc get packagemanifest -n openshift-marketplace
    NAME CATALOG AGE
    etcd My Operator Catalog 34s

    You can also view them from the OperatorHub page in the web console.

You can now install the Operators from the OperatorHub page on your restricted network OpenShift Container Platform cluster.

Updating an Operator catalog image

After a cluster administrator has configured OperatorHub to use custom Operator catalog images, administrators can keep their OpenShift Container Platform cluster up to date with the latest Operators by capturing updates made to Red Hat’s App Registry catalogs. This is done by building and pushing a new Operator catalog image, then replacing the existing CatalogSource’s spec.image parameter with the new image digest.

For this example, the procedure assumes a custom redhat-operators catalog image is already configured for use with OperatorHub.

Prerequisites
  • A Linux workstation with unrestricted network access [1]

  • oc version 4.3.5+

  • podman version 1.4.4+

  • Access to mirror registry that supports Docker v2-2

  • OperatorHub configured to use custom catalog images

  • If you are working with private registries, set the REG_CREDS environment variable to the file path of your registry credentials for use in later steps. For example, for the podman CLI:

    $ REG_CREDS=${XDG_RUNTIME_DIR}/containers/auth.json
Procedure
  1. On the workstation with unrestricted network access, authenticate with the target mirror registry:

    $ podman login <registry_host_name>

    Also authenticate with registry.redhat.io so that the base image can be pulled during the build:

    $ podman login registry.redhat.io
  2. Build a new catalog image based on the redhat-operators catalog from quay.io, tagging and pushing it to your mirror registry:

    $ oc adm catalog build \
        --appregistry-org redhat-operators \(1)
        --from=registry.redhat.io/openshift4/ose-operator-registry:v4.4 \(2)
        --filter-by-os="linux/amd64" \(3)
        --to=<registry_host_name>:<port>/olm/redhat-operators:v2 \(4)
        [-a ${REG_CREDS}] \(5)
        [--insecure] (6)
    
    INFO[0013] loading Bundles                               dir=/var/folders/st/9cskxqs53ll3wdn434vw4cd80000gn/T/300666084/manifests-829192605
    ...
    Pushed sha256:f73d42950021f9240389f99ddc5b0c7f1b533c054ba344654ff1edaf6bf827e3 to example_registry:5000/olm/redhat-operators:v2
    1 Organization (namespace) to pull from an App Registry instance.
    2 Set --from to the ose-operator-registry base image using the tag that matches the target OpenShift Container Platform cluster major and minor version.
    3 Set --filter-by-os to the operating system and architecture to use for the base image, which must match the target OpenShift Container Platform cluster. Valid values are linux/amd64, linux/ppc64le, and linux/s390x.
    4 Name your catalog image and include a tag, for example, v2 because it is the updated catalog.
    5 Optional: If required, specify the location of your registry credentials file.
    6 Optional: If you do not want to configure trust for the target registry, add the --insecure flag.
  3. Mirror the contents of your catalog to your target registry. The following oc adm catalog mirror command extracts the contents of your custom Operator catalog image to generate the manifests required for mirroring and mirrors the images to your registry:

    $ oc adm catalog mirror \
        <registry_host_name>:<port>/olm/redhat-operators:v2 \(1)
        <registry_host_name>:<port> \
        [-a ${REG_CREDS}] \(2)
        [--insecure] \(3)
        [--filter-by-os="<os>/<arch>"] (4)
    
    mirroring ...
    1 Specify your new Operator catalog image.
    2 Optional: If required, specify the location of your registry credentials file.
    3 Optional: If you do not want to configure trust for the target registry, add the --insecure flag.
    4 Optional: Because the catalog might reference images that support multiple architectures and operating systems, you can filter by architecture and operating system to mirror only the images that match. Valid values are linux/amd64, linux/ppc64le, and linux/s390x.
  4. Apply the newly generated manifests:

    $ oc apply -f ./redhat-operators-manifests

    It is possible that you do not need to apply the imageContentSourcePolicy.yaml manifest. Complete a diff of the files to determine if changes are necessary.

  5. Update your CatalogSource object that references your catalog image.

    1. If you have your original catalogsource.yaml file for this CatalogSource:

      1. Edit your catalogsource.yaml file to reference your new catalog image in the spec.image field:

        apiVersion: operators.coreos.com/v1alpha1
        kind: CatalogSource
        metadata:
          name: my-operator-catalog
          namespace: openshift-marketplace
        spec:
          sourceType: grpc
          image: <registry_host_name>:<port>/olm/redhat-operators:v2 (1)
          displayName: My Operator Catalog
          publisher: grpc
        1 Specify your new Operator catalog image.
      2. Use the updated file to replace the CatalogSource object:

        $ oc replace -f catalogsource.yaml
    2. Alternatively, edit the CatalogSource using the following command and reference your new catalog image in the spec.image parameter:

      $ oc edit catalogsource <catalog_source_name> -n openshift-marketplace

Updated Operators should now be available from the OperatorHub page on your OpenShift Container Platform cluster.

Testing an Operator catalog image

You can validate Operator catalog image content by running it as a container and querying its gRPC API. To further test the image, you can then resolve an OLM Subscription by referencing the image in a CatalogSource. For this example, the procedure uses a custom redhat-operators catalog image previously built and pushed to a supported registry.

Prerequisites
  • A custom Operator catalog image pushed to a supported registry

  • podman version 1.4.4+

  • oc version 4.3.5+

  • Access to mirror registry that supports Docker v2-2

  • grpcurl

Procedure
  1. Pull the Operator catalog image:

    $ podman pull <registry_host_name>:<port>/olm/redhat-operators:v1
  2. Run the image:

    $ podman run -p 50051:50051 \
        -it <registry_host_name>:<port>/olm/redhat-operators:v1
  3. Query the running image for available packages using grpcurl:

    $ grpcurl -plaintext localhost:50051 api.Registry/ListPackages
    {
      "name": "3scale-operator"
    }
    {
      "name": "amq-broker"
    }
    {
      "name": "amq-online"
    }
  4. Get the latest Operator bundle in a channel:

    $  grpcurl -plaintext -d '{"pkgName":"kiali-ossm","channelName":"stable"}' localhost:50051 api.Registry/GetBundleForChannel
    {
      "csvName": "kiali-operator.v1.0.7",
      "packageName": "kiali-ossm",
      "channelName": "stable",
    ...
  5. Get the digest of the image:

    $ podman inspect \
        --format='{{index .RepoDigests 0}}' \
        <registry_host_name>:<port>/olm/redhat-operators:v1
    
    example_registry:5000/olm/redhat-operators@sha256:f73d42950021f9240389f99ddc5b0c7f1b533c054ba344654ff1edaf6bf827e3
  6. Assuming an OperatorGroup exists in namespace my-ns that supports your Operator and its dependencies, create a CatalogSource object using the image digest. For example:

    apiVersion: operators.coreos.com/v1alpha1
    kind: CatalogSource
    metadata:
      name: custom-redhat-operators
      namespace: my-ns
    spec:
      sourceType: grpc
      image: example_registry:5000/olm/redhat-operators@sha256:f73d42950021f9240389f99ddc5b0c7f1b533c054ba344654ff1edaf6bf827e3
      displayName: Red Hat Operators
  7. Create a Subscription that resolves the latest available servicemeshoperator and its dependencies from your catalog image:

    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: servicemeshoperator
      namespace: my-ns
    spec:
      source: custom-redhat-operators
      sourceNamespace: my-ns
      name: servicemeshoperator
      channel: "1.0"

1. The oc adm catalog command is currently only supported on Linux. (BZ#1771329)