OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history
×

Performing Automated Cluster Upgrades

Overview

Starting with OpenShift 3.0.2, if you installed using the advanced installation and the inventory file that was used is available, you can use the upgrade playbook to automate the OpenShift cluster upgrade process. If you installed using the quick installation method and a ~/.config/openshift/installer.cfg.yml file is available, you can use the installer to perform the automated upgrade.

The automated upgrade performs the following steps for you:

  • Applies the latest configuration.

  • Upgrades and restart master services.

  • Upgrades and restart node services.

  • Applies the latest cluster policies.

  • Updates the default router if one exists.

  • Updates the default registry if one exists.

  • Updates default image streams and InstantApp templates.

Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

Running Ansible playbooks with the --tags or --check options is not supported by Red Hat.

Preparing for an Automated Upgrade

If you are on OpenShift Enterprise 3.0, you must first upgrade to 3.1 before upgrading to 3.2. Further, if you are currently using the Pacemaker HA method, you must first upgrade to the native HA method before upgrading to 3.2, as the Pacemaker method is no longer supported starting with 3.2. See the OpenShift Enterprise 3.1 upgrade documentation for instructions.

Starting with RHBA-2016:1208, upgrades from OpenShift Enterprise 3.1 to 3.2 are supported for clusters using the containerized installation method. See Known Issues.

To prepare for an automated upgrade:

  1. If you are upgrading from OpenShift Enterprise 3.1 to 3.2, on each master and node host you must manually disable the 3.1 channel and enable the 3.2 channel:

    # subscription-manager repos --disable="rhel-7-server-ose-3.1-rpms" \
    --enable="rhel-7-server-ose-3.2-rpms"\
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms"
# yum clean all

  2. For any upgrade path, always ensure that you have the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible-* packages:

    # yum update atomic-openshift-utils

  3. Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Enterprise version:

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  4. You must be logged in as a cluster administrative user on the master host for the upgrade to succeed:

    $ oc login

There are two methods for running the automated upgrade: using the installer or running the upgrade playbook directly. Choose and follow one method.

Using the Installer to Upgrade

If you installed OpenShift Enterprise using the quick installation method, you should have an installation configuration file located at ~/.config/openshift/installer.cfg.yml. The installer requires this file to start an upgrade.

The installer supports upgrading between minor versions of OpenShift Enterprise (e.g., 3.1 to 3.2) as well as between asynchronous errata updates within a minor version (e.g., 3.2.z).

If you have an older format installation configuration file in ~/.config/openshift/installer.cfg.yml from an existing OpenShift Enterprise 3.0 or 3.1 installation, the installer will attempt to upgrade the file to the new supported format. If you do not have an installation configuration file of any format, you can create one manually.

To start the upgrade, run the installer with the upgrade subcommand:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.

  2. Run the following command on each host to remove the atomic-openshift packages from the list of yum excludes on the host:

    # atomic-openshift-excluder unexclude

  3. Run the installer with the upgrade subcommand:

    # atomic-openshift-installer upgrade

  4. Follow the on-screen instructions to upgrade to the latest release.

  5. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. Before rebooting, run the following command on each master and node host to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude

    Then reboot all hosts.

  6. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    Feature Next Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

Running the Upgrade Playbook Directly

You can run the automated upgrade playbook using Ansible directly, similar to the advanced installation method, if you have an inventory file.

The same v3_2 upgrade playbook can be used to upgrade either of the following to the latest 3.2 release:

Upgrading to OpenShift Enterprise 3.2

Before running the upgrade, first ensure the deployment_type parameter in your inventory file is set to openshift-enterprise.

If you have multiple masters configured and want to enable rolling, full system restarts of the hosts, you can set the openshift_rolling_restart_mode parameter in your inventory file to system. Otherwise, the default value services performs rolling service restarts on HA masters, but does not reboot the systems. See Configuring Cluster Variables for details.

Then, run the v3_2 upgrade playbook. If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/.ansible/hosts for the last inventory file that was used, if needed.

# ansible-playbook [-i </path/to/inventory/file>] \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_2/upgrade.yml

The upgrade playbook was previously located in a v3_1_to_v3_2 directory. Ensure you are using the latest playbooks per the Preparing for an Automated Upgrade section.

  1. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. Before rebooting, run the following command on each master and node host to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude

    Then reboot all hosts.

  2. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    Feature Next Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

Upgrading to OpenShift Enterprise 3.2 Asynchronous Releases

To apply asynchronous errata updates to an existing OpenShift Enterprise 3.2 cluster, first upgrade the atomic-openshift-utils package on the Red Hat Enterprise Linux 7 system where you will be running Ansible:

# yum update atomic-openshift-utils

Then, run the same v3_2 upgrade playbook that is used for upgrading to OpenShift Enterprise 3.2 from 3.1. If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify the location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/.ansible/hosts for the last inventory file that was used, if needed.

# ansible-playbook [-i </path/to/inventory/file>] \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_2/upgrade.yml

  1. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. Before rebooting, run the following command on each master and node host to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude

    Then reboot all hosts.

  2. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    Feature Next Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

Upgrading the EFK Logging Stack

If you have previously deployed the EFK logging stack and want to upgrade to the latest logging component images, the steps must be performed manually as shown in Manual Upgrades.

Upgrading Cluster Metrics

If you have previously deployed cluster metrics, you must manually update to the latest metric components.

Verifying the Upgrade

To verify the upgrade, first check that all nodes are marked as Ready:

# oc get nodes
NAME                 LABELS                                                                STATUS
master.example.com   kubernetes.io/hostname=master.example.com,region=infra,zone=default   Ready
node1.example.com    kubernetes.io/hostname=node1.example.com,region=primary,zone=east     Ready

Then, verify that you are running the expected versions of the docker-registry and router images, if deployed:

# oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:v3.2.1.31",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:v3.2.1.31",

After upgrading, you can use the diagnostics tool on the master to look for common issues:

# oadm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.