Soon, all Red Hat OpenShift documentation will only be available on docs.redhat.com, the official location of all Red Hat product documentation. Get familiar with the updated experience today!
  1. Documentation
    2. history
×

Performing Automated In-place Cluster Upgrades

Overview

An etcd performance issue has been discovered on new and upgraded OpenShift Container Platform 3.5 clusters. See the following Knowledgebase Solution for further details:

https://access.redhat.com/solutions/2916381
(BZ#1415839)

If you installed using the advanced installation and the inventory file that was used is available, you can use the upgrade playbook to automate the OpenShift cluster upgrade process. If you installed using the quick installation method and a ~/.config/openshift/installer.cfg.yml file is available, you can use the installer to perform the automated upgrade.

The automated upgrade performs the following steps for you:

  • Applies the latest configuration.

  • Upgrades master and etcd components and restarts services.

  • Upgrades node components and restarts services.

  • Applies the latest cluster policies.

  • Updates the default router if one exists.

  • Updates the default registry if one exists.

  • Updates default image streams and InstantApp templates.

Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

Running Ansible playbooks with the --tags or --check options is not supported by Red Hat.

Preparing for an Automated Upgrade

Before upgrading your cluster to OpenShift Container Platform 3.5, the cluster must be already upgraded to the latest asynchronous release of version 3.4. Cluster upgrades cannot span more than one minor version at a time, so if your cluster is at a version earlier than 3.4, you must first upgrade incrementally (e.g., 3.2 to 3.3, then 3.3 to 3.4).

Before attempting the upgrade, follow the steps in Verifying the Upgrade to verify the cluster’s health. This will confirm that nodes are in the Ready state, running the expected starting version, and will ensure that there are no diagnostic errors or warnings.

To prepare for an automated upgrade:

  1. If you are upgrading from OpenShift Container Platform 3.4 to 3.5, manually disable the 3.4 channel and enable the 3.5 channel on each master and node host:

    # subscription-manager repos --disable="rhel-7-server-ose-3.4-rpms" \
    --enable="rhel-7-server-ose-3.5-rpms" \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-fast-datapath-rpms"
# yum clean all

  2. For any upgrade path, always ensure that you have the latest version of the atomic-openshift-utils package on each RHEL 7 system, which also updates the openshift-ansible-* packages:

    # yum update atomic-openshift-utils

  3. Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Container Platform version:

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

    1. When installing or updating atomic-openshift-utils, /usr/share/openshift/examples/ does not get updated with the latest templates. To update the file:

      # mkdir /usr/share/openshift/examples
# cp -R /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/* /usr/share/openshift/examples/

To persist /usr/share/openshift/examples/ on all masters:

mkdir /usr/share/openshift/examples
scp -R /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/* user@masterx:/usr/share/openshift/examples

You must be logged in as a cluster administrative user on the master host for the upgrade to succeed:

$ oc login

After satisfying these steps, there are two methods for running the automated upgrade:

Choose and follow one of these methods.

Using the Installer to Upgrade

If you installed OpenShift Container Platform using the quick installation method, you should have an installation configuration file located at ~/.config/openshift/installer.cfg.yml. The installer requires this file to start an upgrade.

The installer supports upgrading between minor versions of OpenShift Container Platform (one minor version at a time, e.g., 3.4 to 3.5) as well as between asynchronous errata updates within a minor version (e.g., 3.5.z).

If you have an older format installation configuration file in ~/.config/openshift/installer.cfg.yml from an installation of a previous cluster version, the installer will attempt to upgrade the file to the new supported format. If you do not have an installation configuration file of any format, you can create one manually.

To start an upgrade with the quick installer:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.

  2. Run the installer with the upgrade subcommand:

    # atomic-openshift-installer upgrade

  3. Then, follow the on-screen instructions to upgrade to the latest release.

  4. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    Feature Next Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

Running Upgrade Playbooks Directly

You can run automated upgrade playbooks using Ansible directly, similar to the advanced installation method, if you have an inventory file. Playbooks can be run using the ansible-playbook command.

The same v3_5 upgrade playbooks can be used for either of the following scenarios:

  • Upgrading existing OpenShift Container Platform 3.4 clusters to 3.5

  • Upgrading existing OpenShift Container Platform 3.5 clusters to the latest asynchronous errata updates

Upgrading the Control Plane and Nodes in Separate Phases

An OpenShift Container Platform cluster can be upgraded in one or more phases. You can choose whether to upgrade all hosts in one phase by running a single Ansible playbook, or upgrade the control plane (master components) and nodes in multiple phases using separate playbooks.

Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OpenShift Container Platform 3.5 Release.

When upgrading in separate phases, the control plane phase includes upgrading:

  • etcd

  • master components

  • node services running on masters

  • Docker running on masters

  • Docker running on any separate etcd hosts

When upgrading only the nodes, the control plane must already be upgraded. The node phase includes upgrading:

  • node services running on stand-alone nodes

  • Docker running on stand-alone nodes

Nodes running master components are not included during the node upgrade phase, even though they have node services and Docker running on them. Instead, they are upgraded as part of the control plane upgrade phase. This ensures node services and Docker on masters are not upgraded twice (once during the control plane phase and again during the node phase).

Customizing Node Upgrades

Whether upgrading in a single or multiple phases, you can customize how the node portion of the upgrade progresses by passing certain Ansible variables to an upgrade playbook using the -e option.

Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OpenShift Container Platform 3.5 Release.

The openshift_upgrade_nodes_serial variable can be set to an integer or percentage to control how many node hosts are upgraded at the same time. The default is 1, upgrading nodes one at a time.

For example, to upgrade 20 percent of the total number of detected nodes at a time:

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="20%"

The openshift_upgrade_nodes_label variable allows you to specify that only nodes with a certain label are upgraded. This can also be combined with the openshift_upgrade_nodes_serial variable.

For example, to only upgrade nodes in the group1 region, two at a time:

$ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="2" \
    -e openshift_upgrade_nodes_label="region=group1"

See Managing Nodes for more on node labels.

Customizing Upgrades With Ansible Hooks

When upgrading OpenShift Container Platform, you can execute custom tasks during specific operations through a system called hooks. Hooks allow cluster administrators to provide files defining tasks to execute before and/or after specific areas during installations and upgrades. This can be very helpful to validate or modify custom infrastructure when installing or upgrading OpenShift Container Platform.

It is important to remember that when a hook fails, the operation fails. This means a good hook can run multiple times and provide the same results. A great hook is idempotent.

Limitations

  • Hooks have no defined or versioned interface. They can use internal openshift-ansible variables, but there is no guarantee these will remain in future releases. In the future, hooks may be versioned, giving you advance warning that your hook needs to be updated to work with the latest openshift-ansible.

  • Hooks have no error handling, so an error in a hook will halt the upgrade process. The problem will need to be addressed and the upgrade re-run.

Using Hooks

Hooks are defined in the hosts inventory file under the OSEv3:vars section.

Each hook must point to a YAML file which defines Ansible tasks. This file will be used as an include, meaning that the file cannot be a playbook, but a set of tasks. Best practice suggests using absolute paths to the hook file to avoid any ambiguity.

Example Hook Definitions in an Inventory File
[OSEv3:vars]
openshift_master_upgrade_pre_hook=/usr/share/custom/pre_master.yml
openshift_master_upgrade_hook=/usr/share/custom/master.yml
openshift_master_upgrade_post_hook=/usr/share/custom/post_master.yml
Example pre_master.yml Task
---
# Trivial example forcing an operator to ack the start of an upgrade
# file=/usr/share/custom/pre_master.yml

- name: note the start of a master upgrade
  debug:
      msg: "Master upgrade of {{ inventory_hostname }} is about to start"

- name: require an operator agree to start an upgrade
  pause:
      prompt: "Hit enter to start the master upgrade"

Available Upgrade Hooks

openshift_master_upgrade_pre_hook

  • Runs before each master is upgraded.

  • This hook runs against each master in serial.

  • If a task must run against a different host, said task must use delegate_to or local_action.

openshift_master_upgrade_hook

  • Runs after each master is upgraded, but before its service or system restart.

  • This hook runs against each master in serial.

  • If a task must run against a different host, said task must use delegate_to or local_action.

openshift_master_upgrade_post_hook

  • Runs after each master is upgraded and has had its service or system restart.

  • This hook runs against each master in serial.

  • If a task must run against a different host, said task must use delegate_to or local_action.

Upgrading to the Latest OpenShift Container Platform 3.5 Release

To upgrade an existing OpenShift Container Platform 3.4 or 3.5 cluster to the latest 3.5 release:

  1. Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.

  2. Ensure the deployment_type parameter in your inventory file is set to openshift-enterprise.

  3. If you have multiple masters configured and want to enable rolling, full system restarts of the hosts, you can set the openshift_rolling_restart_mode parameter in your inventory file to system. Otherwise, the default value services performs rolling service restarts on HA masters, but does not reboot the systems. See Configuring Cluster Variables for details.

  4. At this point, you can choose to run the upgrade in a single or multiple phases. See Upgrading the Control Plane and Nodes in Separate Phases for more details which components are upgraded in each phase.

    If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify its location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/hosts for the last inventory file that was used, if needed.

    You can add --tags pre_upgrade to the following ansible-playbook commands to run the pre-upgrade checks for the playbook. This is a dry-run option that preforms all pre-upgrade checks without actually upgrading any hosts, and reports any problems found.

    • Option A) Upgrade control plane and nodes in a single phase.

      Run the upgrade.yml playbook to upgrade the cluster in a single phase using one playbook; the control plane is still upgraded first, then nodes in-place:

      # ansible-playbook -i </path/to/inventory/file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_5/upgrade.yml \
    [-e <customized_node_upgrade_variables>] (1)
      1 See Customizing Node Upgrades for any desired <customized_node_upgrade_variables>.

    • Option B) Upgrade the control plane and nodes in separate phases.

      1. To upgrade only the control plane, run the upgrade_control_plane.yaml playbook:

        # ansible-playbook -i </path/to/inventory/file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_5/upgrade_control_plane.yml

      2. To upgrade only the nodes, run the upgrade_nodes.yaml playbook:

        # ansible-playbook -i </path/to/inventory/file> \
    /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_5/upgrade_nodes.yml \
    [-e <customized_node_upgrade_variables>] (1)
        1 See Customizing Node Upgrades for any desired <customized_node_upgrade_variables>.

        If you are upgrading the nodes in groups as described in Customizing Node Upgrades, continue invoking the upgrade_nodes.yml playbook until all nodes have been successfully upgraded.

  5. After all master and node upgrades have completed, a recommendation will be printed to reboot all hosts. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features have you previously enabled.

    Feature Next Step

    Aggregated Logging

    Upgrade the EFK logging stack.

    Cluster Metrics

    Upgrade cluster metrics.

Upgrading the EFK Logging Stack

To upgrade an existing EFK logging stack deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-logging.yml Ansible playbook. This is the playbook to use if you were deploying logging for the first time on an existing cluster, but is also used to upgrade existing logging deployments.

  1. If you have not already done so, see Specifying Logging Ansible Variables in the Aggregating Container Logs topic and update your Ansible inventory file to at least set the following required variable within the [OSEv3:vars] section:

    [OSEv3:vars]

openshift_hosted_logging_deploy=true (1)
openshift_hosted_logging_deployer_prefix=registry.example.com:8888/openshift3/ (2)
openshift_hosted_logging_deployer_version=3.5.0 (3)
    1 Enables the ability to upgrade the logging stack.
    2 Replace registry.example.com:8888/openshift3/ with your desired prefix.
    3 Replace with the desired image version, or with v3.5 for the latest version.

  2. Add any other openshift_logging_* or openshift_hosted_logging_* variables that you want to specify to override the defaults, as described in Specifying Logging Ansible Variables.

    +

See the OpenShift Container Platform example Ansible host file in the openshift/openshift-ansible repo for the full list of variables.

  1. When you have finished updating your inventory file, follow the instructions in Deploying the EFK Stack to run the openshift-logging.yml playbook and complete the logging deployment upgrade.

Upgrading Cluster Metrics

To upgrade an existing cluster metrics deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml Ansible playbook. This is the playbook to use if you were deploying metrics for the first time on an existing cluster, but is also used to upgrade existing metrics deployments.

  1. If you have not already done so, see Specifying Metrics Ansible Variables in the Enabling Cluster Metrics topic and update your Ansible inventory file to at least set the following required variables within the [OSEv3:vars] section:

    [OSEv3:vars]

openshift_metrics_install_metrics=true (1)
openshift_metrics_image_version=<tag> (2)
openshift_metrics_hawkular_hostname=<fqdn> (3)
openshift_metrics_cassandra_storage_type=(emptydir|pv|dynamic) (4)
    1 Enables the ability to upgrade the metrics deployment.
    2 Replace <tag> with v3.5 for the latest version.
    3 Used for the Hawkular Metrics route. Should correspond to a fully qualified domain name.
    4 Choose a type that is consistent with the previous deployment.

  2. Add any other openshift_metrics_* or openshift_hosted_metrics_* variables that you want to specify to override the defaults, as described in Specifying Metrics Ansible Variables.

    +

See the OpenShift Container Platform example Ansible host file in the openshift/openshift-ansible repo for the full list of variables.

  1. When you have finished updating your inventory file, follow the instructions in Deploying the Metrics Deployment to run the openshift_metrics.yml playbook and complete the metrics deployment upgrade.

Verifying the Upgrade

To verify the upgrade:

  1. If you are using a proxy, you must add the IP address of the etcd endpoints to the openshift_no_proxy cluster variable in your inventory file.

    If you are not using a proxy, you can skip this step.

    In OpenShift Container Platform 3.4, the master connected to the etcd cluster using the host name of the etcd endpoints. In OpenShift Container Platform 3.5, the master now connects to etcd via IP address.

    When configuring a cluster to use proxy settings (see Configuring Global Proxy Options), this change causes the master-to-etcd connection to be proxied as well, rather than being excluded by host name in each host’s NO_PROXY setting (see Working with HTTP Proxies for more about NO_PROXY).

    1. To workaround this issue, add the IP address of the etcd endpoint to the NO_PROXY environment variable on each master host’s /etc/sysconfig/atomic-openshift-master-controllers file. For example:

      NO_PROXY=<ip_address>

      Use the IP address or host name that the master uses to contact the etcd cluster as the <ip_address>. The <port> should be 2379 if you are using standalone etcd (clustered) or 4001 for embedded etcd (single master, non-clustered etcd). The installer will be updated in a future release to handle this scenario automatically during installation and upgrades (BZ#1466783).

    2. Restart the master service for the changes to take effect:

      # systemctl restart atomic-openshift-master

  2. Check that all nodes are marked as Ready:

    # oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

  3. Then, verify that you are running the expected versions of the docker-registry and router images, if deployed. Replace <tag> with v3.5.5.31.66 for the latest version.

    # oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:<tag>",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:<tag>",

  4. You can use the diagnostics tool on the master to look for common issues:

    # oc adm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.