Soon, all Red Hat OpenShift documentation will only be available on docs.redhat.com, the official location of all Red Hat product documentation. Get familiar with the updated experience today!
  1. Documentation
    2. history
×

Performing Manual In-place Cluster Upgrades

Overview

An etcd performance issue has been discovered on new and upgraded OpenShift Container Platform 3.6 clusters. See the following Knowledgebase Solution for further details:

https://access.redhat.com/solutions/2916381
(BZ#1415839)

As an alternative to performing an automated upgrade, you can manually upgrade your OpenShift cluster. To manually upgrade without disruption, it is important to upgrade each component as documented in this topic.

Before you begin your upgrade, familiarize yourself now with the entire procedure. Specific releases may require additional steps to be performed at key points before or during the standard upgrade process.

Ensure that you have met all prerequisites before proceeding with an upgrade. Failure to do so can result in a failed upgrade.

The day before the upgrade, validate OpenShift Container Platform storage migration to ensure potential issues are resolved prior to the outage window:

$ oc adm migrate storage --include=* --loglevel=2 --confirm --config /etc/origin/master/admin.kubeconfig

Preparing for a Manual Upgrade

Before upgrading your cluster to OpenShift Container Platform 3.6, the cluster must be already upgraded to the latest asynchronous release of version 3.5. Cluster upgrades cannot span more than one minor version at a time, so if your cluster is at a version earlier than 3.5, you must first upgrade incrementally (e.g., 3.3 to 3.4, then 3.4 to 3.5).

Before attempting the upgrade, follow the steps in Verifying the Upgrade to verify the cluster’s health. This will confirm that nodes are in the Ready state, running the expected starting version, and will ensure that there are no diagnostic errors or warnings.

To prepare for a manual upgrade, follow these steps:

  1. If you are upgrading from OpenShift Container Platform 3.5 to 3.6, manually disable the 3.5 channel and enable the 3.6 channel on each host:

    # subscription-manager repos --disable="rhel-7-server-ose-3.5-rpms" \
    --enable="rhel-7-server-ose-3.6-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-fast-datapath-rpms"

    On RHEL 7 systems, also clear the yum cache:

    # yum clean all

  2. Install or update to the latest available version of the atomic-openshift-utils package on each RHEL 7 system, which provides files that will be used in later sections:

    # yum install atomic-openshift-utils

    1. When installing or updating atomic-openshift-utils, /usr/share/openshift/examples/ does not get updated with the latest templates. To update the file:

      # mkdir /usr/share/openshift/examples
# cp -R /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/* /usr/share/openshift/examples/

  3. Install or update to the following latest available *-excluder packages on each RHEL 7 system, which helps ensure your systems stay on the correct versions of atomic-openshift and docker packages when you are not trying to upgrade, according to the OpenShift Container Platform version:

    # yum install atomic-openshift-excluder atomic-openshift-docker-excluder

    These packages add entries to the exclude directive in the host’s /etc/yum.conf file.

  4. Create an etcd backup on each master. The etcd package is required, even if using embedded etcd, for access to the etcdctl command to make the backup.

    The etcd package is installed by default for RHEL Atomic Host 7 systems. If the master is a RHEL 7 system and etcd is not already installed, install it now:

    # yum install etcd

    To create the backup, run:

    # ETCD_DATA_DIR=/var/lib/origin/openshift.local.etcd (1)
# etcdctl backup \
    --data-dir $ETCD_DATA_DIR \
    --backup-dir $ETCD_DATA_DIR.bak.<date> (2)
    1 This directory is for embedded etcd. If you use a separate etcd cluster, use /var/lib/etcd instead.
    2 Use the date of the backup, or some unique identifier, for <date>. The command will not make a backup if the --backup-dir location already exists.

  5. For any upgrade path, ensure that you are running the latest kernel on each RHEL 7 system:

    # yum update kernel

Upgrading Master Components

Before upgrading any stand-alone nodes, upgrade the master components (which provide the control plane for the cluster).

  1. Run the following command on each master to remove the atomic-openshift packages from the list of yum excludes on the host:

    # atomic-openshift-excluder unexclude

  2. Upgrade etcd on all master hosts and any separate etcd hosts that are not also master hosts.

    1. For RHEL 7 systems using the RPM-based method:

      1. Upgrade the etcd package:

        # yum update etcd

      2. Restart the etcd service and review the logs to ensure it restarts successfully:

        # systemctl restart etcd
# journalctl -r -u etcd

    2. For RHEL Atomic Host 7 systems and RHEL 7 systems using the containerized method:

      1. Pull the latest rhel7/etcd image:

        # docker pull registry.access.redhat.com/rhel7/etcd

      2. Restart the etcd_container service and review the logs to ensure it restarts successfully:

        # systemctl restart etcd_container
# journalctl -r -u etcd_container

  3. On each master host, upgrade the atomic-openshift packages or related images.

    1. For masters using the RPM-based method on a RHEL 7 system, upgrade all installed atomic-openshift packages and the openvswitch package:

      # yum upgrade atomic-openshift\* openvswitch

    2. For masters using the containerized method on a RHEL 7 or RHEL Atomic Host 7 system, set the IMAGE_VERSION parameter to the version you are upgrading to in the following files:

      • /etc/sysconfig/atomic-openshift-master-controllers

      • /etc/sysconfig/atomic-openshift-master-api

      • /etc/sysconfig/atomic-openshift-node

      • /etc/sysconfig/atomic-openshift-openvswitch

      For example:

      IMAGE_VERSION=<tag>

      Replace <tag> with v3.6.173.0.164 for the latest version.

  4. Configure dnsmasq: Because of the changes made in OpenShift Container Platform 3.6, you must perform the following steps to configure dnsmasq as part of the upgrade. See DNS Changes under Notable Technical Changes in the OpenShift Container Platform 3.6 Release Notes.

    1. Download the following script:

      # wget https://raw.githubusercontent.com/openshift/openshift-ansible/release-3.6/roles/openshift_node_dnsmasq/files/networkmanager/99-origin-dns.sh -O /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
# chmod 755 /etc/NetworkManager/dispatcher.d/99-origin-dns.sh

      This command downloads the 99-origin-dns.sh script and copies the script to the /etc/NetworkManager/dispatcher.d/ directory. This script configures pods to use the node IP address as resolver. Using 127.0.0.1 inside a pod would fail.

    2. Create two node-dnsmasq.conf node configuration files. One at the path /etc/origin/node/node-dnsmasq.conf and another at the path /etc/dnsmasq.d/node-dnsmasq.conf. Both these file should have identical content as following:

      server=/in-addr.arpa/127.0.0.1
server=/cluster.local/127.0.0.1

      Sample node-dnsmasq.conf files:

      $ cat /etc/dnsmasq.d/node-dnsmasq.conf
  server=/in-addr.arpa/127.0.0.1
  server=/cluster.local/127.0.0.1
$ cat /etc/origin/node/node-dnsmasq.conf
  server=/in-addr.arpa/127.0.0.1
  server=/cluster.local/127.0.0.1

    3. Edit the /etc/dnsmasq.d/origin-dns.conf file as follows:

      no-resolv
domain-needed
no-negcache
max-cache-ttl=1
enable-dbus
bind-interfaces
listen-address=<node_ip_address> (1)
      1 This is the IP address of the node host.

      Sample origin-dns.conf file:

      $ cat /etc/dnsmasq.d/origin-dns.conf
  no-resolv
  domain-needed
  no-negcache
  max-cache-ttl=1
  enable-dbus
  bind-interfaces
  listen-address=192.0.2.34

    4. Edit the /etc/dnsmasq.d/origin-upstream-dns.conf file as follows:

      server=<upstream-dns-server-1>
server=<upstream-dns-server-2>

      Sample origin-upstream-dns.conf file:

      $ cat /etc/dnsmasq.d/origin-upstream-dns.conf
  server=8.8.8.8
  server=9.9.9.9

    5. Edit the /etc/origin/node/node-config.yaml as follows:

      dnsBindAddress: 127.0.0.1:53
dnsRecursiveResolvConf: /etc/origin/node/resolv.conf
dnsDomain: cluster.local
dnsIP: <node_ip_address> (1)
      1 This is the IP address of the node host.

      Sample node-config.yaml and resolv.conf files:

      $ cat /etc/origin/node/node-config.yaml
      ...
    dnsBindAddress: 127.0.0.1:53
    dnsRecursiveResolvConf: /etc/origin/node/resolv.conf
    dnsDomain: cluster.local
    dnsIP: 192.0.2.34
    dockerConfig:
      ...

$ cat /etc/origin/node/resolv.conf
  nameserver 8.8.8.8
  nameserver 9.9.9.9

    6. Edit the master DNS configuration file to listen on port 8053. This avoids conflicts on port 53 and opens port 8053 in the firewall.

    7. Restart Network Manager:

      # systemctl restart NetworkManager

    8. Edit the /etc/origin/node/node-config.yaml file to set the dnsIP field to the IP address of this node.

  5. Restart the master service(s) on each master and review logs to ensure they restart successfully.

    # systemctl restart atomic-openshift-master-controllers
# systemctl restart atomic-openshift-master-api
# journalctl -r -u atomic-openshift-master-controllers
# journalctl -r -u atomic-openshift-master-api

  6. Because masters also have node components running on them in order to be configured as part of the OpenShift SDN, restart the atomic-openshift-node and openvswitch services:

    # systemctl restart openvswitch
# systemctl restart atomic-openshift-node
# journalctl -r -u openvswitch
# journalctl -r -u atomic-openshift-node

  7. If you are performing a cluster upgrade that requires updating Docker to version 1.12, you must also perform the following steps if you are not already on Docker 1.12:

    The node component on masters is set by default to unschedulable status during initial installation, so that pods are not deployed to them. However, it is possible to set them schedulable during the initial installation or manually thereafter. If any of your masters are also configured as a schedulable node, skip the following Docker upgrade steps for those masters and instead run all steps described in Upgrading Nodes when you get to that section for those hosts as well.

    1. Upgrade the docker package.

      1. For RHEL 7 systems:

        # yum update docker

        Then, restart the docker service and review the logs to ensure it restarts successfully:

        # systemctl restart docker
# journalctl -r -u docker

      2. For RHEL Atomic Host 7 systems, upgrade to the latest Atomic tree if one is available:

        If upgrading to RHEL Atomic Host 7.3.2, this upgrades Docker to version 1.12.

        		 
        # atomic host upgrade

    2. After the upgrade is completed and prepared for the next boot, reboot the host and ensure the docker service starts successfully:

      # systemctl reboot
# journalctl -r -u docker

    3. Remove the following file, which is no longer required:

      # rm /etc/systemd/system/docker.service.d/docker-sdn-ovs.conf

  8. Run the following command on each master to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude

During the cluster upgrade, it can sometimes be useful to take a master out of rotation since some DNS client libraries will not properly to the other masters for cluster DNS. In addition to stopping the master and controller services, you can remove the EndPoint from the Kubernetes service’s subsets.addresses.

$ oc edit ep/kubernetes -n default

When the master is restarted, the Kubernetes service will be automatically updated.

Updating Policy Definitions

After a cluster upgrade, the recommended default cluster roles may be updated. To check if an update is recommended for your environment, you can run:

# oc adm policy reconcile-cluster-roles

If you have customized default cluster roles and want to ensure a role reconciliation does not modify those customized roles, annotate them with openshift.io/reconcile-protect set to true. Doing so means you are responsible for manually updating those roles with any new or required permissions during upgrades.

This command outputs a list of roles that are out of date and their new proposed values. For example:

# oc adm policy reconcile-cluster-roles
apiVersion: v1
items:
- apiVersion: v1
  kind: ClusterRole
  metadata:
    creationTimestamp: null
    name: admin
  rules:
  - attributeRestrictions: null
    resources:
    - builds/custom
...

Your output will vary based on the OpenShift version and any local customizations you have made. Review the proposed policy carefully.

You can either modify this output to re-apply any local policy changes you have made, or you can automatically apply the new policy using the following process:

  1. Reconcile the cluster roles:

    # oc adm policy reconcile-cluster-roles \
    --additive-only=true \
    --confirm

  2. Reconcile the cluster role bindings:

    # oc adm policy reconcile-cluster-role-bindings \
    --exclude-groups=system:authenticated \
    --exclude-groups=system:authenticated:oauth \
    --exclude-groups=system:unauthenticated \
    --exclude-users=system:anonymous \
    --additive-only=true \
    --confirm

    Also run:

    # oc adm policy reconcile-cluster-role-bindings \
    system:build-strategy-jenkinspipeline \
    --confirm \
    -o name

  3. Reconcile security context constraints:

    # oc adm policy reconcile-sccs \
    --additive-only=true \
    --confirm

Upgrading Nodes

After upgrading your masters, you can upgrade your nodes. When restarting the atomic-openshift-node service, there will be a brief disruption of outbound network connectivity from running pods to services while the service proxy is restarted. The length of this disruption should be very short and scales based on the number of services in the entire cluster.

You can alternatively use the blue-green deployment method at this point to create a parallel environment for new nodes instead of upgrading them in place.

One at at time for each node that is not also a master, you must disable scheduling and evacuate its pods to other nodes, then upgrade packages and restart services.

  1. Run the following command on each node to remove the atomic-openshift packages from the list of yum excludes on the host:

    # atomic-openshift-excluder unexclude

  2. As a user with cluster-admin privileges, disable scheduling for the node:

    # oc adm manage-node <node> --schedulable=false

  3. Evacuate pods on the node to other nodes:

    The --force option deletes any pods that are not backed by a replication controller.

    		 
    # oc adm drain <node> --force --delete-local-data --ignore-daemonsets

  4. Upgrade the node component packages or related images.

    1. For nodes using the RPM-based method on a RHEL 7 system, upgrade all installed atomic-openshift packages and the openvswitch package:

      # yum upgrade atomic-openshift\* openvswitch

    2. For nodes using the containerized method on a RHEL 7 or RHEL Atomic Host 7 system, set the IMAGE_VERSION parameter in the /etc/sysconfig/atomic-openshift-node and /etc/sysconfig/openvswitch files to the version you are upgrading to. For example:

      IMAGE_VERSION=<tag>

      Replace <tag> with v3.6.173.0.164 for the latest version.

  5. Configure dnsmasq: Because of the changes made in OpenShift Container Platform 3.6, you must perform the following steps to configure dnsmasq as part of the upgrade. See DNS Changes under Notable Technical Changes in the OpenShift Container Platform 3.6 Release Notes.

    1. Download the following script:

      # wget https://raw.githubusercontent.com/openshift/openshift-ansible/release-3.6/roles/openshift_node_dnsmasq/files/networkmanager/99-origin-dns.sh -O /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
# chmod 755 /etc/NetworkManager/dispatcher.d/99-origin-dns.sh

      This command downloads the 99-origin-dns.sh script and copies the script to the /etc/NetworkManager/dispatcher.d/ directory. This script configures pods to use the node IP address as resolver. Using 127.0.0.1 inside a pod would fail.

    2. Create two node-dnsmasq.conf node configuration files. One at the path /etc/origin/node/node-dnsmasq.conf and another at the path /etc/dnsmasq.d/node-dnsmasq.conf. Both these file should have identical content as following:

      server=/in-addr.arpa/127.0.0.1
server=/cluster.local/127.0.0.1

      Sample node-dnsmasq.conf files:

      $ cat /etc/dnsmasq.d/node-dnsmasq.conf
  server=/in-addr.arpa/127.0.0.1
  server=/cluster.local/127.0.0.1
$ cat /etc/origin/node/node-dnsmasq.conf
  server=/in-addr.arpa/127.0.0.1
  server=/cluster.local/127.0.0.1

    3. Edit the /etc/dnsmasq.d/origin-dns.conf file as follows:

      no-resolv
domain-needed
no-negcache
max-cache-ttl=1
enable-dbus
bind-interfaces
listen-address=<node_ip_address> (1)
      1 This is the IP address of the node host.

      Sample origin-dns.conf file:

      $ cat /etc/dnsmasq.d/origin-dns.conf
  no-resolv
  domain-needed
  no-negcache
  max-cache-ttl=1
  enable-dbus
  bind-interfaces
  listen-address=192.0.2.34

    4. Edit the /etc/dnsmasq.d/origin-upstream-dns.conf file as follows:

      server=<upstream-dns-server-1>
server=<upstream-dns-server-2>

      Sample origin-upstream-dns.conf file:

      $ cat /etc/dnsmasq.d/origin-upstream-dns.conf
  server=8.8.8.8
  server=9.9.9.9

    5. Edit the /etc/origin/node/node-config.yaml as follows:

      dnsBindAddress: 127.0.0.1:53
dnsRecursiveResolvConf: /etc/origin/node/resolv.conf
dnsDomain: cluster.local
dnsIP: <node_ip_address> (1)
      1 This is the IP address of the node host.

      Sample node-config.yaml and resolv.conf files:

      $ cat /etc/origin/node/node-config.yaml
      ...
    dnsBindAddress: 127.0.0.1:53
    dnsRecursiveResolvConf: /etc/origin/node/resolv.conf
    dnsDomain: cluster.local
    dnsIP: 192.0.2.34
    dockerConfig:
      ...

$ cat /etc/origin/node/resolv.conf
  nameserver 8.8.8.8
  nameserver 9.9.9.9

    6. Edit the master DNS configuration file to listen on port 8053. This avoids conflicts on port 53 and opens port 8053 in the firewall.

    7. Restart Network Manager:

      # systemctl restart NetworkManager

    8. Edit the /etc/origin/node/node-config.yaml file to set the dnsIP field to the IP address of this node.

  6. Restart the atomic-openshift-node and openvswitch services and review the logs to ensure they restart successfully:

    # systemctl restart openvswitch
# systemctl restart atomic-openshift-node
# journalctl -r -u atomic-openshift-node
# journalctl -r -u openvswitch

  7. If you are performing a cluster upgrade that requires updating Docker to version 1.12, you must also perform the following steps if you are not already on Docker 1.12:

    1. Upgrade the docker package.

      1. For RHEL 7 systems:

        # yum update docker

        Then, restart the docker service and review the logs to ensure it restarts successfully:

        # systemctl restart docker
# journalctl -r -u docker

        After Docker is restarted, restart the atomic-openshift-node service again and review the logs to ensure it restarts successfully:

        # systemctl restart atomic-openshift-node
# journalctl -r -u atomic-openshift-node

      2. For RHEL Atomic Host 7 systems, upgrade to the latest Atomic tree if one is available:

        If upgrading to RHEL Atomic Host 7.3.2, this upgrades Docker to version 1.12.

        		 
        # atomic host upgrade

        After the upgrade is completed and prepared for the next boot, reboot the host and ensure the docker service starts successfully:

        # systemctl reboot
# journalctl -r -u docker

    2. Remove the following file, which is no longer required:

      # rm /etc/systemd/system/docker.service.d/docker-sdn-ovs.conf

  8. Re-enable scheduling for the node:

    # oc adm manage-node <node> --schedulable

  9. Run the following command on each node to add the atomic-openshift packages back to the list of yum excludes on the host:

    # atomic-openshift-excluder exclude

  10. Repeat the previous steps on the next node, and continue repeating these steps until all nodes have been upgraded.

  11. After all nodes have been upgraded, as a user with cluster-admin privileges, verify that all nodes are showing as Ready:

    # oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

Upgrading the Router

If you have previously deployed a router, the router deployment configuration must be upgraded to apply updates contained in the router image. To upgrade your router without disrupting services, you must have previously deployed a highly-available routing service.

Edit your router’s deployment configuration. For example, if it has the default router name:

# oc edit dc/router

Apply the following changes:

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-haproxy-router:<tag> (1)
        imagePullPolicy: IfNotPresent
        ...
1 Adjust <tag> to match the version you are upgrading to (use v3.6.173.0.164 for the latest version).

You should see one router pod updated and then the next.

Upgrading the Registry

The registry must also be upgraded for changes to take effect in the registry image. If you have used a PersistentVolumeClaim or a host mount point, you may restart the registry without losing the contents of your registry. Storage for the Registry details how to configure persistent storage for the registry.

Edit your registry’s deployment configuration:

# oc edit dc/docker-registry

Apply the following changes:

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/ose-docker-registry:<tag> (1)
        imagePullPolicy: IfNotPresent
        ...
1 Adjust <tag> to match the version you are upgrading to (use v3.6.173.0.164 for the latest version).

If the registry console is deployed, edit its deployment configuration:

# oc edit dc/registry-console

Apply the following changes:

...
spec:
 template:
    spec:
      containers:
      - env:
        ...
        image: registry.access.redhat.com/openshift3/registry-console:v3.6
        imagePullPolicy: IfNotPresent
        ...

Images that are being pushed or pulled from the internal registry at the time of upgrade will fail and should be restarted automatically. This will not disrupt pods that are already running.

Updating the Default Image Streams and Templates

By default, the quick and advanced installation methods automatically create default image streams, InstantApp templates, and database service templates in the openshift project, which is a default project to which all users have view access. These objects were created during installation from the JSON files located under the /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/ directory.

Because RHEL Atomic Host 7 cannot use yum to update packages, the following steps must take place on a RHEL 7 system.

Update the packages that provide the example JSON files. On a subscribed Red Hat Enterprise Linux 7 system where you can run the CLI as a user with cluster-admin permissions, install or update to the latest version of the atomic-openshift-utils package, which should also update the openshift-ansible- packages:

# yum update atomic-openshift-utils

To persist /usr/share/openshift/examples/ on the first master:

scp -R /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/* user@master1:/usr/share/openshift/examples/

To persist /usr/share/openshift/examples/ on all masters:

mkdir /usr/share/openshift/examples
scp -R /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/* user@masterx:/usr/share/openshift/examples

The openshift-ansible-roles package provides the latest example JSON files.

  1. After a manual upgrade, get the latest templates from openshift-ansible-roles:

    rpm -ql openshift-ansible-roles | grep examples | grep v3.6

    In this example, /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/image-streams/image-streams-rhel7.json is the latest file that you want in the latest openshift-ansible-roles package.

    /usr/share/openshift/examples/image-streams/image-streams-rhel7.json is not owned by a package, but is updated by Ansible. If you are upgrading outside of Ansible. you need to get the latest .json files on the system where you are running oc, which can run anywhere that has access to the master.

  2. Install atomic-openshift-utils and its dependencies to install the new content into /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/.:

    $ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/image-streams/image-streams-rhel7.json
$ oc create -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/image-streams/dotnet_imagestreams.json
$ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/image-streams/image-streams-rhel7.json
$ oc replace -n openshift -f  /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/image-streams/dotnet_imagestreams.json

  3. Update the templates:

    $ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/quickstart-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/db-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/infrastructure-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/xpaas-templates/
$ oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/xpaas-streams/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/quickstart-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/db-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/infrastructure-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/xpaas-templates/
$ oc replace -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/xpaas-streams/

    Errors are generated for items that already exist. This is expected behavior:

    # oc create -n openshift -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/quickstart-templates/
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/quickstart-templates/cakephp-mysql.json": templates "cakephp-mysql-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/quickstart-templates/cakephp.json": templates "cakephp-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/quickstart-templates/dancer-mysql.json": templates "dancer-mysql-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/quickstart-templates/dancer.json": templates "dancer-example" already exists
Error from server: error when creating "/usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v3.6/quickstart-templates/django-postgresql.json": templates "django-psql-example" already exists

Now, content can be updated. Without running the automated upgrade playbooks, the content is not updated in /usr/share/openshift/.

Importing the Latest Images

After updating the default image streams, you may also want to ensure that the images within those streams are updated. For each image stream in the default openshift project, you can run:

# oc import-image -n openshift <imagestream>

For example, get the list of all image streams in the default openshift project:

# oc get is -n openshift
NAME     DOCKER REPO                                                      TAGS                   UPDATED
mongodb  registry.access.redhat.com/openshift3/mongodb-24-rhel7           2.4,latest,v3.1.1.6    16 hours ago
mysql    registry.access.redhat.com/openshift3/mysql-55-rhel7             5.5,latest,v3.1.1.6    16 hours ago
nodejs   registry.access.redhat.com/openshift3/nodejs-010-rhel7           0.10,latest,v3.1.1.6   16 hours ago
...

Update each image stream one at a time:

# oc import-image -n openshift nodejs
The import completed successfully.

Name:			nodejs
Created:		10 seconds ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2016-07-05T19:20:30Z
Docker Pull Spec:	172.30.204.22:5000/openshift/nodejs

Tag	Spec								Created		PullSpec						Image
latest	4								9 seconds ago	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest	570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
4	registry.access.redhat.com/rhscl/nodejs-4-rhel7:latest		9 seconds ago	<same>							570ad8ed927fd5c2c9554ef4d9534cef808dfa05df31ec491c0969c3bd372b05
0.10	registry.access.redhat.com/openshift3/nodejs-010-rhel7:latest	9 seconds ago	<same>							a1ef33be788a28ec2bdd48a9a5d174ebcfbe11c8e986d2996b77f5bccaaa4774

In order to update your S2I-based applications, you must manually trigger a new build of those applications after importing the new images using oc start-build <app-name>.

Upgrading the EFK Logging Stack

Manual upgrade steps for logging deployments are no longer available starting in OpenShift Container Platform 3.5.

To upgrade an existing EFK logging stack deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-logging.yml Ansible playbook. This is the playbook to use if you were deploying logging for the first time on an existing cluster, but is also used to upgrade existing logging deployments.

  1. If you have not already done so, see Specifying Logging Ansible Variables in the Aggregating Container Logs topic and update your Ansible inventory file to at least set the following required variable within the [OSEv3:vars] section:

    [OSEv3:vars]

openshift_hosted_logging_deploy=true (1)
openshift_hosted_logging_deployer_prefix=registry.example.com:8888/openshift3/ (2)
openshift_hosted_logging_deployer_version=3.6 (3)
    1 Enables the ability to upgrade the logging stack.
    2 Replace registry.example.com:8888/openshift3/ with your desired prefix.
    3 Replace 3.6 with v3.6 for the latest version if desired.

  2. Add any other openshift_logging_* or openshift_hosted_logging_* variables that you want to override the defaults, as described in Specifying Logging Ansible Variables.

    See the OpenShift Container Platform example Ansible host file in the openshift/openshift-ansible repo for the full list of variables.

  3. When you have finished updating your inventory file, follow the instructions in Deploying the EFK Stack to run the openshift-logging.yml playbook and complete the logging deployment upgrade.

If your Fluentd DeploymentConfig and DaemonSet for the EFK components are already set with:

        image: <image_name>:<vX.Y>
        imagePullPolicy: IfNotPresent

The latest version <image_name> might not be pulled if there is already one with the same <image_name:vX.Y> stored locally on the node where the pod is being re-deployed. If so, manually change the DeploymentConfig and DaemonSet to imagePullPolicy: Always to make sure it is re-pulled.

Upgrading Cluster Metrics

Manual upgrade steps for metrics deployments are no longer available starting in OpenShift Container Platform 3.5.

To upgrade an existing cluster metrics deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/openshift-metrics.yml Ansible playbook. This is the playbook to use if you were deploying metrics for the first time on an existing cluster, but is also used to upgrade existing metrics deployments.

  1. If you have not already done so, see Specifying Metrics Ansible Variables in the Enabling Cluster Metrics topic and update your Ansible inventory file to at least set the following required variables within the [OSEv3:vars] section:

    [OSEv3:vars]

openshift_metrics_install_metrics=true (1)
openshift_metrics_image_version=<tag> (2)
openshift_metrics_hawkular_hostname=<fqdn> (3)
openshift_metrics_cassandra_storage_type=(emptydir|pv|dynamic) (4)
    1 Enables the ability to upgrade the metrics deployment.
    2 Replace <tag> with v3.6 for the latest version.
    3 Used for the Hawkular Metrics route. Should correspond to a fully qualified domain name.
    4 Choose a type that is consistent with the previous deployment.

  2. Add any other openshift_metrics_* or openshift_hosted_metrics_* variables that you want to override the defaults, as described in Specifying Metrics Ansible Variables.

    See the OpenShift Container Platform example Ansible host file in the openshift/openshift-ansible repo for the full list of variables.

  3. When you have finished updating your inventory file, follow the instructions in Deploying the Metrics Deployment to run the openshift_metrics.yml playbook and complete the metrics deployment upgrade.

Additional Manual Steps Per Release

Some OpenShift Container Platform releases may have additional instructions specific to that release that must be performed to fully apply the updates across the cluster. This section will be updated over time as new asynchronous updates are released for OpenShift Container Platform 3.6.

See the OpenShift Container Platform 3.6 Release Notes to review the latest release notes.

Verifying the Upgrade

To verify the upgrade:

  1. Check that all nodes are marked as Ready:

    # oc get nodes
NAME                        STATUS                     AGE
master.example.com          Ready,SchedulingDisabled   165d
node1.example.com           Ready                      165d
node2.example.com           Ready                      165d

  2. Verify that you are running the expected versions of the docker-registry and router images, if deployed. Replace <tag> with v3.6.173.0.164 for the latest version.

    # oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:<tag>",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:<tag>",

  3. Use the diagnostics tool on the master to look for common issues:

    # oc adm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.