Disconnected Installation - Installing a Cluster | Installation and Configuration

Overview

Frequently, portions of a datacenter may not have access to the Internet, even via proxy servers. Installing OpenShift Container Platform in these environments is considered a disconnected installation.

An OpenShift Container Platform disconnected installation differs from a regular installation in two primary ways:

The OpenShift Container Platform software channels and repositories are not available via Red Hat’s content distribution network.
OpenShift Container Platform uses several containerized components. Normally, these images are pulled directly from Red Hat’s Docker registry. In a disconnected environment, this is not possible.

A disconnected installation ensures the OpenShift Container Platform software is made available to the relevant servers, then follows the same installation process as a standard connected installation. This topic additionally details how to manually download the container images and transport them onto the relevant servers.

Once installed, in order to use OpenShift Container Platform, you will need source code in a source control repository (for example, Git). This topic assumes that an internal Git repository is available that can host source code and this repository is accessible from the OpenShift Container Platform nodes. Installing the source control repository is outside the scope of this document.

Also, when building applications in OpenShift Container Platform, your build may have some external dependencies, such as a Maven Repository or Gem files for Ruby applications. For this reason, and because they might require certain tags, many of the Quickstart templates offered by OpenShift Container Platform may not work on a disconnected environment. However, while Red Hat container images try to reach out to external repositories by default, you can configure OpenShift Container Platform to use your own internal repositories. For the purposes of this document, we assume that such internal repositories already exist and are accessible from the OpenShift Container Platform nodes hosts. Installing such repositories is outside the scope of this document.

You can also have a Red Hat Satellite server that provides access to Red Hat content via an intranet or LAN. For environments with Satellite, you can synchronize the OpenShift Container Platform software onto the Satellite for use with the OpenShift Container Platform servers.

Red Hat Satellite 6.1 also introduces the ability to act as a Docker registry, and it can be used to host the OpenShift Container Platform containerized components. Doing so is outside of the scope of this document.

Prerequisites

This document assumes that you understand OpenShift Container Platform’s overall architecture and that you have already planned out what the topology of your environment will look like.

Required Software and Components

In order to pull down the required software repositories and container images, you will need a Red Hat Enterprise Linux (RHEL) 7 server with access to the Internet and at least 100GB of additional free space. All steps in this section should be performed on the Internet-connected server as the root system user.

Syncing Repositories

Before you sync with the required repositories, you may need to import the appropriate GPG key:

# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release

If the key is not imported, the indicated package is deleted after syncing the repository.

To sync the required repositories:

Register the server with the Red Hat Customer Portal. You must use the login and password associated with the account that has access to the OpenShift Container Platform subscriptions:
```
# subscription-manager register
```

Attach to a subscription that provides OpenShift Container Platform channels. You can find the list of available subscriptions using:

# subscription-manager list --available --matches '*OpenShift*'

Then, find the pool ID for the subscription that provides OpenShift Container Platform, and attach it:

# subscription-manager attach --pool=<pool_id>
# subscription-manager repos --disable="*"
# subscription-manager repos \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-fast-datapath-rpms" \
    --enable="rhel-7-server-ose-3.5-rpms"

The yum-utils command provides the reposync utility, which lets you mirror yum repositories, and createrepo can create a usable yum repository from a directory:
```
# yum -y install yum-utils createrepo docker git
```
You will need up to 110GB of free space in order to sync the software. Depending on how restrictive your organization’s policies are, you could re-connect this server to the disconnected LAN and use it as the repository server. You could use USB-connected storage and transport the software to another server that will act as the repository server. This topic covers these options.
Make a path to where you want to sync the software (either locally or on your USB or other device):
```
# mkdir -p </path/to/repos>
```

Sync the packages and create the repository for each of them. You will need to modify the command for the appropriate path you created above:

# for repo in \
rhel-7-server-rpms \
rhel-7-server-extras-rpms \
rhel-7-fast-datapath-rpms \
rhel-7-server-ose-3.5-rpms
do
  reposync --gpgcheck -lm --repoid=${repo} --download_path=/path/to/repos
  createrepo -v </path/to/repos/>${repo} -o </path/to/repos/>${repo}
done

Syncing Images

To sync the container images:

Start the Docker daemon:
```
# systemctl start docker
```

Pull all of the required OpenShift Container Platform containerized components. Replace <tag> with v3.5.5.31.66 for the latest version.

# docker pull registry.access.redhat.com/openshift3/ose-f5-router:<tag>
# docker pull registry.access.redhat.com/openshift3/ose-deployer:<tag>
# docker pull registry.access.redhat.com/openshift3/ose:<tag>
# docker pull registry.access.redhat.com/openshift3/ose-egress-router:<tag>
# docker pull registry.access.redhat.com/openshift3/ose-keepalived-ipfailover:<tag>
# docker pull registry.access.redhat.com/openshift3/openvswitch:<tag>
# docker pull registry.access.redhat.com/openshift3/ose-recycler:<tag>
# docker pull registry.access.redhat.com/openshift3/ose-docker-builder:<tag>
# docker pull registry.access.redhat.com/openshift3/ose-docker-registry:<tag>
# docker pull registry.access.redhat.com/openshift3/ose-haproxy-router:<tag>
# docker pull registry.access.redhat.com/openshift3/node:<tag>
# docker pull registry.access.redhat.com/openshift3/ose-pod:<tag>
# docker pull registry.access.redhat.com/openshift3/ose-sti-builder:<tag>

If you are using NFS, you need the ose-recycler image. Otherwise, the volumes will not recycle, potentially causing errors.

Pull all of the required OpenShift Container Platform containerized components for the additional centralized log aggregation and metrics aggregation components. Replace <tag> with v3.5 for the latest version.

# docker pull registry.access.redhat.com/openshift3/logging-curator:<tag>
# docker pull registry.access.redhat.com/openshift3/logging-elasticsearch:<tag>
# docker pull registry.access.redhat.com/openshift3/logging-kibana:<tag>
# docker pull registry.access.redhat.com/openshift3/metrics-deployer:<tag>
# docker pull registry.access.redhat.com/openshift3/metrics-hawkular-openshift-agent:<tag>
# docker pull registry.access.redhat.com/openshift3/logging-auth-proxy:<tag>
# docker pull registry.access.redhat.com/openshift3/logging-deployer:<tag>
# docker pull registry.access.redhat.com/openshift3/logging-fluentd:<tag>
# docker pull registry.access.redhat.com/openshift3/metrics-cassandra:<tag>
# docker pull registry.access.redhat.com/openshift3/metrics-hawkular-metrics:<tag>
# docker pull registry.access.redhat.com/openshift3/metrics-heapster:<tag>

Pull the Red Hat-certified Source-to-Image (S2I) builder images that you intend to use in your OpenShift environment. You can pull the following images:

# docker pull registry.access.redhat.com/jboss-amq-6/amq63-openshift
# docker pull registry.access.redhat.com/jboss-datagrid-7/datagrid71-openshift
# docker pull registry.access.redhat.com/jboss-datagrid-7/datagrid71-client-openshift
# docker pull registry.access.redhat.com/jboss-datavirt-6/datavirt63-openshift
# docker pull registry.access.redhat.com/jboss-datavirt-6/datavirt63-driver-openshift
# docker pull registry.access.redhat.com/jboss-decisionserver-6/decisionserver64-openshift
# docker pull registry.access.redhat.com/jboss-processserver-6/processserver64-openshift
# docker pull registry.access.redhat.com/jboss-eap-6/eap64-openshift
# docker pull registry.access.redhat.com/jboss-eap-7/eap70-openshift
# docker pull registry.access.redhat.com/jboss-webserver-3/webserver31-tomcat7-openshift
# docker pull registry.access.redhat.com/jboss-webserver-3/webserver31-tomcat8-openshift
# docker pull registry.access.redhat.com/openshift3/jenkins-1-rhel7
# docker pull registry.access.redhat.com/openshift3/jenkins-2-rhel7
# docker pull registry.access.redhat.com/openshift3/jenkins-slave-base-rhel7
# docker pull registry.access.redhat.com/openshift3/jenkins-slave-maven-rhel7
# docker pull registry.access.redhat.com/openshift3/jenkins-slave-nodejs-rhel7
# docker pull registry.access.redhat.com/rhscl/mongodb-32-rhel7
# docker pull registry.access.redhat.com/rhscl/mysql-57-rhel7
# docker pull registry.access.redhat.com/openshift3/jenkins-slave-nodejs-rhel7
# docker pull registry.access.redhat.com/rhscl/perl-524-rhel7
# docker pull registry.access.redhat.com/rhscl/php-56-rhel7
# docker pull registry.access.redhat.com/rhscl/postgresql-95-rhel7
# docker pull registry.access.redhat.com/rhscl/python-35-rhel7
# docker pull registry.access.redhat.com/redhat-sso-7/sso70-openshift
# docker pull registry.access.redhat.com/rhscl/ruby-24-rhel7
# docker pull registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift
# docker pull registry.access.redhat.com/redhat-sso-7/sso71-openshift
# docker pull registry.access.redhat.com/rhscl/nodejs-4-rhel7
# docker pull registry.access.redhat.com/rhscl/mariadb-101-rhel7

Make sure to indicate the correct tag specifying the desired version number. For example, to pull both the previous and latest version of the Tomcat image:

# docker pull \
registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:latest
# docker pull \
registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:1.1

See the S2I table in the OpenShift and Atomic Platform Tested Integrations page for details about OpenShift image version compatibility.

If you are using a stand-alone registry or plan to enable the registry console with the integrated registry, you must pull the registry-console image.

Replace <tag> with 3.5 for the latest version.
```
# docker pull registry.access.redhat.com/openshift3/registry-console:<tag>
```

Preparing Images for Export

Container images can be exported from a system by first saving them to a tarball and then transporting them:

Make and change into a repository home directory:

# mkdir </path/to/repos/images>
# cd </path/to/repos/images>

Export the OpenShift Container Platform containerized components:

# docker save -o ose3-images.tar \
    registry.access.redhat.com/openshift3/ose-f5-router \
    registry.access.redhat.com/openshift3/ose-deployer \
    registry.access.redhat.com/openshift3/ose \
    registry.access.redhat.com/openshift3/ose-egress-router \
    registry.access.redhat.com/openshift3/ose-keepalived-ipfailover \
    registry.access.redhat.com/openshift3/openvswitch \
    registry.access.redhat.com/openshift3/ose-recycler \
    registry.access.redhat.com/openshift3/ose-docker-builder \
    registry.access.redhat.com/openshift3/ose-docker-registry \
    registry.access.redhat.com/openshift3/ose-haproxy-router \
    registry.access.redhat.com/openshift3/node \
    registry.access.redhat.com/openshift3/ose-pod \
    registry.access.redhat.com/openshift3/ose-sti-builder \
    registry.access.redhat.com/openshift3/registry-console

If you synchronized the metrics and log aggregation images, export:

# docker save -o ose3-logging-metrics-images.tar \
    registry.access.redhat.com/openshift3/logging-curator:<tag>
    registry.access.redhat.com/openshift3/logging-elasticsearch:<tag>
    registry.access.redhat.com/openshift3/logging-kibana:<tag>
    registry.access.redhat.com/openshift3/metrics-deployer:<tag>
    registry.access.redhat.com/openshift3/metrics-hawkular-openshift-agent:<tag>
    registry.access.redhat.com/openshift3/logging-auth-proxy:<tag>
    registry.access.redhat.com/openshift3/logging-deployer:<tag>
    registry.access.redhat.com/openshift3/logging-fluentd:<tag>
    registry.access.redhat.com/openshift3/metrics-cassandra:<tag>
    registry.access.redhat.com/openshift3/metrics-hawkular-metrics:<tag>
    registry.access.redhat.com/openshift3/metrics-heapster:<tag>

Export the S2I builder images that you synced in the previous section. For example, if you synced only the Jenkins and Tomcat images:

# docker save -o ose3-builder-images.tar \
    registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:latest \
    registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:1.1 \
    registry.access.redhat.com/openshift3/jenkins-1-rhel7 \
    registry.access.redhat.com/openshift3/jenkins-2-rhel7 \
    registry.access.redhat.com/openshift3/jenkins-slave-base-rhel7 \
    registry.access.redhat.com/openshift3/jenkins-slave-maven-rhel7 \
    registry.access.redhat.com/openshift3/jenkins-slave-nodejs-rhel7

Repository Server

During the installation (and for later updates, should you so choose), you will need a webserver to host the repositories. RHEL 7 can provide the Apache webserver.

Option 1: Re-configuring as a Web server

If you can re-connect the server where you synchronized the software and images to your LAN, then you can simply install Apache on the server:

# yum install httpd

Skip to Placing the Software.

Option 2: Building a Repository Server

If you need to build a separate server to act as the repository server, install a new RHEL 7 system with at least 110GB of space. On this repository server during the installation, make sure you select the Basic Web Server option.

Placing the Software

If necessary, attach the external storage, and then copy the repository files into Apache’s root folder. Note that the below copy step (cp -a) should be substituted with move (mv) if you are repurposing the server you used to sync:
```
# cp -a /path/to/repos /var/www/html/
# chmod -R +r /var/www/html/repos
# restorecon -vR /var/www/html
```

Add the firewall rules:

# firewall-cmd --permanent --add-service=http
# firewall-cmd --reload

Enable and start Apache for the changes to take effect:
```
# systemctl enable httpd
# systemctl start httpd
```

OpenShift Container Platform Systems

Building Your Hosts

At this point you can perform the initial creation of the hosts that will be part of the OpenShift Container Platform environment. It is recommended to use the latest version of RHEL 7 and to perform a minimal installation. You will also want to pay attention to the other OpenShift Container Platform-specific prerequisites.

Once the hosts are initially built, the repositories can be set up.

Connecting the Repositories

On all of the relevant systems that will need OpenShift Container Platform software components, create the required repository definitions. Place the following text in the /etc/yum.repos.d/ose.repo file, replacing <server_IP> with the IP or host name of the Apache server hosting the software repositories:

[rhel-7-server-rpms]
name=rhel-7-server-rpms
baseurl=http://<server_IP>/repos/rhel-7-server-rpms
enabled=1
gpgcheck=0
[rhel-7-server-extras-rpms]
name=rhel-7-server-extras-rpms
baseurl=http://<server_IP>/repos/rhel-7-server-extras-rpms
enabled=1
gpgcheck=0
[rhel-7-fast-datapath-rpms]
name=rhel-7-fast-datapath-rpms
baseurl=http://<server_IP>/repos/rhel-7-fast-datapath-rpms
enabled=1
gpgcheck=0
[rhel-7-server-ose-3.5-rpms]
name=rhel-7-server-ose-3.5-rpms
baseurl=http://<server_IP>/repos/rhel-7-server-ose-3.5-rpms
enabled=1
gpgcheck=0

Host Preparation

At this point, the systems are ready to continue to be prepared following the OpenShift Container Platform documentation.

Skip the section titled Host Registration and start with Installing Base Packages.

Installing OpenShift Container Platform

Importing OpenShift Container Platform Containerized Components

To import the relevant components, securely copy the images from the connected host to the individual OpenShift Container Platform hosts:

# scp /var/www/html/repos/images/ose3-images.tar root@<openshift_host_name>:
# ssh root@<openshift_host_name> "docker load -i ose3-images.tar"

If you prefer, you could use wget on each OpenShift Container Platform host to fetch the tar file, and then perform the Docker import command locally. Perform the same steps for the metrics and logging images, if you synchronized them.

On the host that will act as an OpenShift Container Platform master, copy and import the builder images:

# scp /var/www/html/images/ose3-builder-images.tar root@<openshift_master_host_name>:
# ssh root@<openshift_master_host_name> "docker load -i ose3-builder-images.tar"

Running the OpenShift Container Platform Installer

You can now choose to follow the quick or advanced OpenShift Container Platform installation instructions in the documentation.

Creating the Internal Docker Registry

You now need to create the internal Docker registry.

If you want to install a stand-alone registry, you must pull the registry-console container image and set deployment_subtype=registry in the inventory file.

Post-Installation Changes

In one of the previous steps, the S2I images were imported into the Docker daemon running on one of the OpenShift Container Platform master hosts. In a connected installation, these images would be pulled from Red Hat’s registry on demand. Since the Internet is not available to do this, the images must be made available in another Docker registry.

OpenShift Container Platform provides an internal registry for storing the images that are built as a result of the S2I process, but it can also be used to hold the S2I builder images. The following steps assume you did not customize the service IP subnet (172.30.0.0/16) or the Docker registry port (5000).

Re-tagging S2I Builder Images

On the master host where you imported the S2I builder images, obtain the service address of your Docker registry that you installed on the master:
```
# export REGISTRY=$(oc get service docker-registry --template='{{.spec.clusterIP}}{{"\n"}}')
```

Next, tag all of the builder images that you synced and exported before pushing them into the OpenShift Container Platform Docker registry. For example, if you synced and exported only the Tomcat image:

# docker tag \
registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:1.1 \
$REGISTRY:5000/openshift/webserver30-tomcat7-openshift:1.1
# docker tag \
registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:latest \
$REGISTRY:5000/openshift/webserver30-tomcat7-openshift:1.2
# docker tag \
registry.access.redhat.com/jboss-webserver-3/webserver30-tomcat7-openshift:latest \
$REGISTRY:5000/openshift/webserver30-tomcat7-openshift:latest

Configuring a Registry Location

If you are using an image registry other than the default at registry.access.redhat.com, specify the desired registry within the /etc/ansible/hosts file.

oreg_url=example.com/openshift3/ose-${component}:${version}
openshift_examples_modify_imagestreams=true

Depending on your registry, you may need to configure:

openshift_docker_additional_registries=example.com
openshift_docker_insecure_registries=example.com

Table 1. Registry Variables
Variable	Purpose
`oreg_url`	Set to the alternate image location. Necessary if you are not using the default registry at `registry.access.redhat.com`.
`openshift_examples_modify_imagestreams`	Set to `true` if pointing to a registry other than the default. Modifies the image stream location to the value of `oreg_url`.
`openshift_docker_additional_registries`	Set `openshift_docker_additional_registries` to add its value in the `add_registry` line in */etc/sysconfig/docker*. With `add_registry`, you can add your own registry to be used for Docker search and Docker pull. Use the `add_registry` option to list a set of registries, each prepended with `--add-registry` flag. The first registry added will be the first registry searched. For example, `add_registry=--add-registry registry.access.redhat.com --add-registry example.com`.
`openshift_docker_insecure_registries`	Set `openshift_docker_insecure_registries` to add its value in the `insecure_registry` line in */etc/sysconfig/docker*. If you have a registry secured with HTTPS but do not have proper certificates distributed, you can tell Docker not to look for full authorization by adding the registry to the `insecure_registry` line and uncommenting it. For example, `insecure_registry—insecure-registry example.com`.

Creating an Administrative User

Pushing the container images into OpenShift Container Platform’s Docker registry requires a user with cluster-admin privileges. Because the default OpenShift Container Platform system administrator does not have a standard authorization token, they cannot be used to log in to the Docker registry.

To create an administrative user:

Create a new user account in the authentication system you are using with OpenShift Container Platform. For example, if you are using local htpasswd-based authentication:
```
# htpasswd -b /etc/openshift/openshift-passwd <admin_username> <password>
```
The external authentication system now has a user account, but a user must log in to OpenShift Container Platform before an account is created in the internal database. Log in to OpenShift Container Platform for this account to be created. This assumes you are using the self-signed certificates generated by OpenShift Container Platform during the installation:
```
# oc login --certificate-authority=/etc/origin/master/ca.crt \
    -u <admin_username> https://<openshift_master_host>:8443
```

Get the user’s authentication token:

# MYTOKEN=$(oc whoami -t)
# echo $MYTOKEN
iwo7hc4XilD2KOLL4V1O55ExH2VlPmLD-W2-JOd6Fko

Modifying the Security Policies

Using oc login switches to the new user. Switch back to the OpenShift Container Platform system administrator in order to make policy changes:
```
# oc login -u system:admin
```
In order to push images into the OpenShift Container Platform Docker registry, an account must have the image-builder security role. Add this to your OpenShift Container Platform administrative user:
```
# oc adm policy add-role-to-user system:image-builder <admin_username>
```
Next, add the administrative role to the user in the openshift project. This allows the administrative user to edit the openshift project, and, in this case, push the container images:
```
# oc adm policy add-role-to-user admin <admin_username> -n openshift
```

Editing the Image Stream Definitions

The openshift project is where all of the image streams for builder images are created by the installer. They are loaded by the installer from the /usr/share/openshift/examples directory. Change all of the definitions by deleting the image streams which had been loaded into OpenShift Container Platform’s database, then re-create them:

Delete the existing image streams:
```
# oc delete is -n openshift --all
```
Make a backup of the files in /usr/share/openshift/examples/ if you desire. Next, edit the file image-streams-rhel7.json in the /usr/share/openshift/examples/image-streams folder. You will find an image stream section for each of the builder images. Edit the spec stanza to point to your internal Docker registry.

For example, change:
"spec": { "dockerImageRepository": "registry.access.redhat.com/rhscl/mongodb-26-rhel7",
to:
"spec": { "dockerImageRepository": "172.30.69.44:5000/openshift/mongodb-26-rhel7",
In the above, the repository name was changed from rhscl to openshift. You will need to ensure the change, regardless of whether the repository is rhscl, openshift3, or another directory. Every definition should have the following format:
```
<registry_ip>:5000/openshift/<image_name>
```
Repeat this change for every image stream in the file. Ensure you use the correct IP address that you determined earlier. When you are finished, save and exit. Repeat the same process for the JBoss image streams in the /usr/share/openshift/examples/xpaas-streams/jboss-image-streams.json file.

Load the updated image stream definitions:

# oc create -f /usr/share/openshift/examples/image-streams/image-streams-rhel7.json -n openshift
# oc create -f /usr/share/openshift/examples/xpaas-streams/jboss-image-streams.json -n openshift

Loading the Container Images

At this point the system is ready to load the container images.

# docker login -u adminuser -e mailto:adminuser@abc.com \
   -p $MYTOKEN $REGISTRY:5000

Push the Docker images:

# docker push $REGISTRY:5000/openshift/webserver30-tomcat7-openshift:1.1
# docker push $REGISTRY:5000/openshift/webserver30-tomcat7-openshift:1.2
# docker push $REGISTRY:5000/openshift/webserver30-tomcat7-openshift:latest

Verify that all the image streams now have the tags populated:

# oc get imagestreams -n openshift
NAME                                 DOCKER REPO                                                      TAGS                                     UPDATED
jboss-webserver30-tomcat7-openshift  $REGISTRY/jboss-webserver-3/webserver30-jboss-tomcat7-openshift  1.1,1.1-2,1.1-6 + 2 more...              2 weeks ago
...

Installing a Router

At this point, the OpenShift Container Platform environment is almost ready for use. It is likely that you will want to install and configure a router.