OpenShift docs are moving and will soon only be available at docs.redhat.com, the home of all Red Hat product documentation. Explore the new docs experience today.
  1. Documentation
    2. history bug_report picture_as_pdf
×

Performing automated in-place cluster upgrades

If you installed using the standard cluster installation process, and the inventory file that was used is available, you can use upgrade playbooks to automate the cluster upgrade process.

To upgrade OpenShift Container Platform, you run Ansible playbooks with the same inventory file that you used during installation. You run the same v3_11 upgrade playbooks to:

  • Upgrade existing OpenShift Container Platform version 3.10 clusters to version 3.11.

  • Upgrade OpenShift Container Platform version 3.11 clusters to the latest asynchronous errata update.

Running Ansible playbooks with the --tags or --check options is not supported by Red Hat.

Upgrade workflow

The 3.10 to 3.11 control plane upgrade performs the following steps for you:

  • Back up all etcd data for recovery purposes.

  • Update the API and controllers from 3.10 to 3.11.

  • Update internal data structures to 3.11.

  • Update the default router, if one exists, from 3.10 to 3.11.

  • Update the default registry, if one exists, from 3.10 to 3.11.

  • Update the default image streams and InstantApp templates.

The 3.10 to 3.11 node upgrade performs a rolling update of nodes, which:

  • Marks a subset of nodes unschedulable and drains them of pods.

  • Updates node components from 3.10 to 3.11.

  • Returns those nodes to service.

Prerequisites

Before you upgrade your cluster:

  • Review the OpenShift Container Platform 3.11 Release Notes. The release notes contain important notices about changes to OpenShift Container Platform and its function.

  • If you are completing a large-scale upgrade, which involves at least 10 worker nodes and thousands of projects and pods, review Special considerations for large-scale upgrades to prevent upgrade failures.

  • If you are completing a disconnected cluster update, you must update your image registry with new image versions or the cluster update will fail. For example, when updating from 3.11.153 to 3.11.157, make sure the v3.11.157 image tags are present.

  • Upgrade the cluster to the latest asynchronous release of version 3.10. If your cluster is at a version earlier than 3.10, you must first upgrade incrementally. For example, upgrade from 3.7 to 3.9 (the 3.8 version was skipped), and then from 3.9 to 3.10.

  • Run the Environment health checks to verify the cluster’s health. In this process, you confirm that the nodes are in the Ready state and running the expected starting version and that there are no diagnostic errors or warnings.

  • Ensure that your cluster meets the current prerequisites. If it does not, your upgrade might fail.

  • The day before the upgrade, validate OpenShift Container Platform storage migration to ensure potential issues are resolved before the outage window. The following command updates the stored version of API objects for etcd storage:

    $ oc adm migrate storage --include=* --loglevel=2 --config /etc/origin/master/admin.kubeconfig

Preparing for an upgrade

After you satisfy the prerequisites, prepare for an automated upgrade:

  1. Pull the latest subscription data from Red Hat Subscription Manager:

    # subscription-manager refresh

  2. If you are upgrading from OpenShift Container Platform 3.10 to 3.11:

    1. Back up the files that you need if you must downgrade to OpenShift Container Platform 3.10:

      1. On master hosts, back up the following files:

        /etc/origin/master/master-config.yaml
/etc/origin/master/master.env
/etc/origin/master/scheduler.json

      2. On node hosts, including masters, back up the following files:

        /etc/origin/node/node-config.yaml

      3. On etcd hosts, including masters that have etcd co-located on them, back up the following file:

        /etc/etcd/etcd.conf

    2. The upgrade process creates a backup of all etcd data for recovery purposes, but ensure that you have a recent etcd backup at /backup/etcd-xxxxxx/backup.db before continuing. Manual etcd backup steps are described in the Day Two Operations Guide.

      When you upgrade OpenShift Container Platform, your etcd configuration does not change. Whether you run etcd as static pods on master hosts or as a separate service on master hosts or separate hosts does not change after you upgrade.

    3. Manually disable the 3.10 repository and enable the 3.11 repository on each master and node host. You must also enable the rhel-7-server-ansible-2.9-rpms repository, if it is not already enabled:

      • For cloud installations and on-premise installations on x86_64 servers, run the following command:

        # subscription-manager repos \
    --disable="rhel-7-server-ose-3.10-rpms" \
    --disable="rhel-7-server-ansible-2.4-rpms" \
    --enable="rhel-7-server-ose-3.11-rpms" \
    --enable="rhel-7-server-rpms" \
    --enable="rhel-7-server-extras-rpms" \
    --enable="rhel-7-server-ansible-2.9-rpms"
# yum clean all

      • For on-premise installations on IBM POWER8 servers, run the following command:

        # subscription-manager repos \
    --disable="rhel-7-for-power-le-ose-3.10-rpms" \
    --enable="rhel-7-for-power-le-rpms" \
    --enable="rhel-7-for-power-le-extras-rpms" \
    --enable="rhel-7-for-power-le-optional-rpms" \
    --enable="rhel-7-server-ansible-2.9-for-power-le-rpms" \
    --enable="rhel-7-server-for-power-le-rhscl-rpms" \
    --enable="rhel-7-for-power-le-ose-3.11-rpms"
# yum clean all

      • For on-premise installations on IBM POWER9 servers, run the following command:

        # subscription-manager repos \
    --disable="rhel-7-for-power-le-ose-3.10-rpms" \
    --enable="rhel-7-for-power-9-rpms" \
    --enable="rhel-7-for-power-9-extras-rpms" \
    --enable="rhel-7-for-power-9-optional-rpms" \
    --enable="rhel-7-server-ansible-2.9-for-power-9-rpms" \
    --enable="rhel-7-server-for-power-9-rhscl-rpms" \
    --enable="rhel-7-for-power-9-ose-3.11-rpms"
# yum clean all

    4. Ensure that you have the latest version of the openshift-ansible package on the host you run the upgrade playbooks on:

      # yum update -y openshift-ansible

    5. Prepare for the Cluster Monitoring Operator. In version 3.11, the Cluster Monitoring Operator is installed on an infrastructure node by default. If your cluster does not use infrastructure nodes:

      • Add an infrastructure node to your cluster.

      • Disable the Cluster Monitoring Operator by adding openshift_cluster_monitoring_operator_install=false to your inventory file.

      • Specify which node to install the Cluster Monitoring Operator on by marking it with the openshift_cluster_monitoring_operator_node_selector.

    6. If you use the standard OpenShift Container Platform registry, prepare for the change from registry.access.redhat.com to registry.redhat.io. Complete the configuration steps in Accessing and Configuring the Red Hat Registry.

  3. Review and update your inventory file.

    1. Ensure that any manual configuration changes you made to your master or node configuration files since your last Ansible playbook run, whether that was initial installation or your most recent cluster upgrade, are in the inventory file. For any variables that are relevant to the manual changes you made, apply the equivalent appropriate changes to your inventory files before running the upgrade. Otherwise, your manual changes might be overwritten by default values during the upgrade, which could cause pods to not run properly or other cluster stability issues.

    2. By default, the installer checks to see if your certificates will expire within a year and fails if they will expire within that time. To change the number of days that your certificate is valid, specify a new value for the openshift_certificate_expiry_warning_days parameter. For example, to ensure that your certificates are valid for 180 days, specify openshift_certificate_expiry_warning_days=180.

    3. To skip checking if your certificates will expire, set openshift_certificate_expiry_fail_on_warn=False.

    4. If you made any changes to admissionConfig settings in your master configuration files, review the openshift_master_admission_plugin_config variable in Configuring Your Inventory File. Failure to do so might cause pods to get stuck in Pending state if you had ClusterResourceOverride settings manually configured previously, as described in Configuring Masters for Overcommitment.

    5. If you used the openshift_hostname parameter in versions of OpenShift Container Platform before 3.10, ensure that the openshift_kubelet_name_override parameter is still in your inventory file and set to the value of openshift_hostname that you used in previous versions.

      You must not remove the openshift_kubelet_name_override parameter from your inventory file after you upgrade.

    6. If you manually manage the cluster’s /etc/origin/master/htpasswd file, add openshift_master_manage_htpasswd=false to your inventory file to prevent the upgrade process from overwriting the htpasswd file.

Updating policy definitions

During a cluster upgrade, and on every restart of any master, the default cluster roles are automatically reconciled to restore any missing permissions.

  1. If you customized default cluster roles and want to ensure a role reconciliation does not modify them, protect each role from reconciliation:

    $ oc annotate clusterrole.rbac <role_name> --overwrite rbac.authorization.kubernetes.io/autoupdate=false

    You must manually update the roles that contain this setting to include any new or required permissions after upgrading.

  2. Generate a default bootstrap policy template file:

    $ oc adm create-bootstrap-policy-file --filename=policy.json

    The contents of the file vary based on the OpenShift Container Platform version, but the file contains only the default policies.

  3. Update the policy.json file to include any cluster role customizations.

  4. Use the policy file to automatically reconcile roles and role bindings that are not reconcile protected:

    $ oc auth reconcile -f policy.json

  5. Reconcile security context constraints:

    # oc adm policy reconcile-sccs \
    --additive-only=true \
    --confirm

Upgrade phases

You can upgrade the OpenShift Container Platform cluster in one or more phases. You can choose to upgrade all hosts in one phase by running a single Ansible playbook or upgrade the control plane, or master components, and nodes in multiple phases using separate playbooks.

If your OpenShift Container Platform cluster uses GlusterFS pods, you must perform the upgrade in multiple phases. See Special Considerations When Using Containerized GlusterFS for details on how to upgrade with GlusterFS.

When upgrading in separate phases, the control plane phase includes upgrading:

  • Master components

  • Node services running on masters

  • Docker or CRI-O running on masters

  • Docker or CRI-O running on any stand-alone etcd hosts

If you upgrade only the nodes, you must first upgrade the control plane. The node phase includes upgrading:

  • Node services running on stand-alone nodes

  • Docker or CRI-O running on stand-alone nodes

Nodes that run master components are upgraded only during the control plane upgrade phase. This ensures that the node services and container engines on masters are not upgraded twice, once during the control plane phase and again during the node phase.

Node upgrade parameters

Whether you upgrade in a single or multiple phases, you can customize how the node portion of the upgrade progresses by passing certain Ansible variables to an upgrade playbook using the -e option.

  • Set the openshift_upgrade_nodes_serial variable to an integer or percentage to control how many node hosts are upgraded at the same time. The default is 1, which upgrades one node at a time.

    For example, to upgrade 20 percent of the total number of detected nodes at a time, run:

    $ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="20%"

  • Set the openshift_upgrade_nodes_label to specify that only nodes with a certain label are upgraded.

    For example, to only upgrade nodes in the group1 region, two at a time:

    $ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial="2" \
    -e openshift_upgrade_nodes_label="region=group1"

    See Managing Nodes for more information about node labels.

  • Set the openshift_upgrade_nodes_max_fail_percentage variable to specify how many nodes can fail in each batch of upgrades. If the percentage of failed nodes exceeds your value, the playbook stops the upgrade process.

  • Set the openshift_upgrade_nodes_drain_timeout variable to specify the length of time to wait before marking a node as failed.

    In this example, 10 nodes are upgraded at a time, the upgrade stops if more than 20 percent of the nodes fail, and a node is marked as failed if it takes more than 600 seconds to drain the node:

    $ ansible-playbook -i <path/to/inventory/file> \
    </path/to/upgrade/playbook> \
    -e openshift_upgrade_nodes_serial=10 \
    -e openshift_upgrade_nodes_max_fail_percentage=20 \
    -e openshift_upgrade_nodes_drain_timeout=600

Ansible hooks for upgrades

When upgrading OpenShift Container Platform, you can execute custom tasks during specific operations through a system called hooks. Hooks allow cluster administrators to provide files defining tasks to execute before or after specific areas during upgrades. You can use hooks to validate or modify custom infrastructure when upgrading OpenShift Container Platform.

Because when a hook fails, the operation fail, design hooks that are idempotent, or can run multiple times and provide the same results.

Limitations

  • Hooks have no defined or versioned interface. They can use internal openshift-ansible variables, but there is no guarantee these variables will remain in future releases. In the future, hooks might be versioned, giving you advance warning that your hook needs to be updated to work with the latest openshift-ansible.

  • Hooks have no error handling, so an error in a hook halts the upgrade process. If you get an error, you must address the problem and then start the upgrade again.

  • You can run node upgrade hooks on only nodes, not masters. To run the hooks on masters, you must specify a master hook for those nodes.

Using hooks

You define hooks in the hosts inventory file under the OSEv3:vars section.

Each hook must point to a YAML file that defines Ansible tasks. This file is used as an include, meaning that the file cannot be a playbook, but is a set of tasks. Best practice suggests using absolute paths to the hook file to avoid any ambiguity.

Example hook definitions in an inventory file
[OSEv3:vars]
openshift_master_upgrade_pre_hook=/usr/share/custom/pre_master.yml
openshift_master_upgrade_hook=/usr/share/custom/master.yml
openshift_master_upgrade_post_hook=/usr/share/custom/post_master.yml

openshift_node_upgrade_pre_hook=/usr/share/custom/pre_node.yml
openshift_node_upgrade_hook=/usr/share/custom/node.yml
openshift_node_upgrade_post_hook=/usr/share/custom/post_node.yml
Example pre_master.yml task
---
# Trivial example forcing an operator to ack the start of an upgrade
# file=/usr/share/custom/pre_master.yml

- name: note the start of a master upgrade
  debug:
      msg: "Master upgrade of {{ inventory_hostname }} is about to start"

- name: require an operator agree to start an upgrade
  pause:
      prompt: "Hit enter to start the master upgrade"

Available upgrade hooks

Table 1. Master Upgrade Hooks
Hook name Description

openshift_master_upgrade_pre_hook

  • Runs before each master is upgraded.

  • This hook runs against each master in serial.

  • If a task must run against a different host, the task must use delegate_to or local_action.

openshift_master_upgrade_hook

  • Runs after each master is upgraded but before its service or system restart.

  • This hook runs against each master in serial.

  • If a task must run against a different host, the task must use delegate_to or local_action.

openshift_master_upgrade_post_hook

  • Runs after each master is upgraded and its service or system restarts.

  • This hook runs against each master in serial.

  • If a task must run against a different host, the task must use delegate_to or local_action.
Table 2. Node upgrade hooks
Hook name Description

openshift_node_upgrade_pre_hook

  • Runs before each node is upgraded.

  • This hook runs against each node in serial.

  • If a task must run against a different host, the task must use delegate_to or local_action.

openshift_node_upgrade_hook

  • Runs after each node is upgraded but before it is marked schedulable again.

  • This hook runs against each node in serial.

  • If a task must run against a different host, they task must use delegate_to or local_action.

openshift_node_upgrade_post_hook

  • Runs after each node is upgraded. It is the last node upgrade action.

  • This hook runs against each node in serial.

  • If a task must run against a different host, the task must use delegate_to or local_action.

Special considerations for upgrading OpenShift Container Platform

If your OpenShift Container Platform cluster uses a mixed environment or gcePD storage, you need to take more steps before you upgrade it.

Before you upgrade a mixed environment, such as one with Red Hat Enterprise Linux (RHEL) and RHEL Atomic Host, set values in the inventory file for both the openshift_pkg_version and openshift_image_tag parameters. Setting these values ensures that all nodes in your cluster run the same version of OpenShift Container Platform. While this is a best practice for major updates, such as from OpenShift Container Platform 2 to OpenShift Container Platform 3, setting these values are mandatory for minor version upgrades.

For example, to upgrade from OpenShift Container Platform 3.9 to OpenShift Container Platform 3.10, set the following parameters and values:

openshift_pkg_version=-3.10.16
openshift_image_tag=v3.10.16

These parameters can also be present in other, non-mixed, environments.

Special considerations for large-scale upgrades

For large-scale cluster upgrades, which involve at least 10 worker nodes and thousands of projects and pods, the API object storage migration should be performed prior to running the upgrade playbooks, and then again after the upgrade has successfully completed. Otherwise, the upgrade process will fail.

Refer to the Running the pre- and post- API server model object migration outside of the upgrade window section of the Recommendations for large-scale OpenShift upgrades for further guidance.

Special considerations when using gcePD

Because the default gcePD storage provider uses an RWO (Read-Write Only) access mode, you cannot perform a rolling upgrade on the registry or scale the registry to multiple pods. Therefore, when upgrading OpenShift Container Platform, you must specify the following environment variables in your Ansible inventory file:

[OSEv3:vars]

openshift_hosted_registry_storage_provider=gcs
openshift_hosted_registry_storage_gcs_bucket=bucket01
openshift_hosted_registry_storage_gcs_keyfile=test.key
openshift_hosted_registry_storage_gcs_rootdirectory=/registry

Upgrading to the latest OpenShift Container Platform release

To upgrade an existing OpenShift Container Platform 3.10 or 3.11 cluster to the latest 3.11 release:

  1. Prepare for an upgrade to ensure you use the latest upgrade playbooks.

  2. Ensure the openshift_deployment_type parameter in your inventory file is set to openshift-enterprise.

  3. To enable rolling, full system restarts of the hosts, set the openshift_rolling_restart_mode parameter in your inventory file to system. Otherwise, the service is restarted on masters, but the systems do not reboot.

    The openshift_rolling_restart_mode only works for master hosts.

    See Configuring Cluster Variables for details.

  4. If you modified the oreg_url parameter to change the cluster image registry location, you must run the imageconfig playbook to update the image location:

    $ cd /usr/share/ansible/openshift-ansible
$ ansible-playbook -i </path/to/inventory/file> \
    playbooks/openshift-node/imageconfig.yml

  5. Upgrade your nodes.

    If your inventory file is located somewhere other than the default /etc/ansible/hosts, add the -i flag to specify its location. If you previously used the atomic-openshift-installer command to run your installation, you can check ~/.config/openshift/hosts for the last inventory file that was used.

    • To upgrade control plane and nodes in a single phase, run the upgrade.yml playbook:

      $ cd /usr/share/ansible/openshift-ansible
$ ansible-playbook -i </path/to/inventory/file> \
    playbooks/byo/openshift-cluster/upgrades/v3_11/upgrade.yml

    • To upgrade the control plane and nodes in separate phases:

      1. Upgrade the control plane by running the upgrade_control_plane.yaml playbook:

        $ cd /usr/share/ansible/openshift-ansible
$ ansible-playbook -i </path/to/inventory/file> \
    playbooks/byo/openshift-cluster/upgrades/v3_11/upgrade_control_plane.yml

      2. Upgrade the nodes by running the upgrade_nodes.yaml playbook:

        $ cd /usr/share/ansible/openshift-ansible
$ ansible-playbook -i </path/to/inventory/file> \
    playbooks/byo/openshift-cluster/upgrades/v3_11/upgrade_nodes.yml \
    [-e <customized_node_upgrade_variables>] (1)
        1 See Customizing Node Upgrades for any desired <customized_node_upgrade_variables>.

        If you are upgrading the nodes in groups as described in Customizing Node Upgrades, continue running the upgrade_nodes.yml playbook until all nodes are upgraded.

  6. If you did not enable automated reboot of the master hosts by using openshift_rolling_restart variable=system in step 3 of this procedure, you can manually reboot all master hosts together with all node hosts after the upgrade has completed. Rebooting the hosts is optional.

  7. If you use aggregated logging, upgrade the EFK logging stack.

  8. If you use cluster metrics, upgrade cluster metrics.

  9. Verify the upgrade.

Upgrading OpenShift Container Platform when using containerized GlusterFS

When upgrading OpenShift Container Platform, you must upgrade the set of nodes where GlusterFS pods run. However, because these pods run as part of a daemonset, you cannot use drain or unschedule commands to terminate and evacuate the GlusterFS pods. To avoid data availability and cluster corruption, you must also upgrade nodes that host GlusterFS pods one at a time to ensure that the upgrade process completes on a node that runs GlusterFS before the upgrade starts on the next node.

To upgrade OpenShift Container Platform if you use containerized GlusterFS:

  1. Upgrade the control plane (the master nodes and etcd nodes).

  2. Upgrade standard infra nodes (router, registry, logging, and metrics).

    If any of the nodes in those groups are running GlusterFS, perform step 4 of this procedure at the same time. GlusterFS nodes must be upgraded along with other nodes in their class (app versus infra), one at a time.

  3. Upgrade standard nodes running application containers.

    If any of the nodes in those groups are running GlusterFS, perform step 4 of this procedure at the same time. GlusterFS nodes must be upgraded along with other nodes in their class (app versus infra), one at a time.

  4. Upgrade the OpenShift Container Platform nodes running GlusterFS one at a time.

    1. Add the following parameters in the inventory file at /etc/ansible/hosts:

      openshift_hosted_registry_storage_kind=glusterfs
openshift_storage_glusterfs_heketi_image=registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7:<your_cns_vesion>(1)
openshift_storage_glusterfs_image=registry.access.redhat.com/rhgs3/rhgs-server-rhel7:<your_cns_vesion>(1)
openshift_storage_glusterfs_block_image=registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7:<your_cns_vesion>(1)
openshift_storage_glusterfs_s3_image=registry.access.redhat.com/rhgs3/rhgs-s3-server-rhel7:<your_cns_vesion>(1)
      1 Specify the CNS version, such as v3.9 or v3.11.3.

    2. Update the image tags from latest to <your_cns_version>, such as v3.9 or v3.11.3, on the following resources, if they are present in your configuration:

      $ oc edit -n <glusterfs_namespace> ds glusterfs-<name>
$ oc edit -n <glusterfs_namespace> dc heketi-<name>
$ oc edit -n <glusterfs_namespace> dc glusterblock-<name>-provisioner-dc
$ oc edit -n <glusterfs_namespace> dc gluster-<name>-<account>-s3

    3. Add a label to the node you want to upgrade so that only one node is upgraded at a time:

      $ oc label node <node_name> type=upgrade

    4. Do not terminate the GlusterFS pod you want to restart.

    5. To run the upgrade playbook on a single GlusterFS node, use -e openshift_upgrade_nodes_label="type=upgrade".

      The GlusterFS pod should not be terminated.

    6. Wait for the GlusterFS pod to respawn and appear.

    7. Run basic health checks after each pod restart to ensure they pass.

    8. oc rsh into the pod and verify all volumes are healed:

      $ oc rsh <GlusterFS_pod_name>
$ for vol in `gluster volume list`; do gluster volume heal $vol info; done

      Ensure all of the volumes are healed and there are no outstanding tasks. The heal info command lists all pending entries for a given volume’s heal process. A volume is considered healed when Number of entries for that volume is 0.

    9. Remove the upgrade label and go to the next GlusterFS node.

      $ oc label node <node_name> type-

Upgrading optional components

If you installed an EFK logging stack or cluster metrics, you must separately upgrade the component.

Upgrading the EFK Logging Stack

To upgrade an existing EFK logging stack deployment, you review your parameters and run the openshift-logging/config.yml playbook.

The upgrade can replace your logging-fluentd and logging-curator ConfigMaps. If you want to retain your current logging-fluentd and logging-curator ConfigMaps, set the openshift_logging_fluentd_replace_configmap and openshift_logging_curator_replace_configmap parameters to no in the inventory host file.

The EFK upgrade also upgrades Elasticsearch from version 2 to version 5. For important information on changes in Elasticsearch 5, you should review the Elasticsearch breaking changes.

It is important to note that Elasticsearch 5 has some significant changes to the index structures. Previously, Elasticsearch permitted a dot character, ., in field names. In version 5, Elasticsearch interprets any dot in an Elasticsearch field name as nested structure. If you have a field with a dot, the string after the dot is interpreted as the type of field, leading to mapping conflicts during the upgrade.

To help identify potential conflicts, OpenShift Container Platform provides a script that examines your Elasticsearch fields to determine if any fields contain a dot in the name.

For example, the following fields were allowed in Elasticsearch 2:

{
  "field": 123  // "field" is of type number
}

// Any dot in field name is treated as any other valid character in the field name.
// It is just part of the field name.
{
  "field.name": "Bob"  // "field.name" is of type String

In Elasticsearch 5 and higher the field string would become the field and the name string would become a type for the field:

{
  "field": 123  // "field" is of type number
}

// Any dot in field name is always interpreted as nested structure.
{
  "field" : {  // "field" is of type Object
    "name": "Bob"  // "name" is of type String
  }
}

Upgrading in this case would result in the field field having two different types, which is not permitted.

If you need to keep these conflicting indices, you need to reindex the data and change the documents to get rid of conflicting data structure. For more information, see Upgrading fields with dots to 5.x.

Determining if fields have dots in field names

You can run the following script to determine if your indices contain any fields with a dot in the name.

The following command uses the jq JSON processor to get directly at the necessary data. Red Hat Enterprise Linux (RHEL), depending on version, might not provide a package for jq. You might need to install this from external sources, or unsupported locations.

 
oc exec -c elasticsearch -n $LOGGING_NS $pod -- es_util --query='_mapping?pretty&filter_path=**.mappings.*.properties' \
  | jq '.[].mappings[].properties | keys' \
  | jq .[] \
  | egrep -e "\."

The upgrade path depends on whether the indices have fields with dots or do not have fields with dots.

Upgrading if fields have dots in field names

If the script above indicates your indices contain fields with a dot in the name, use the following steps to correct this issue and upgrade.

To upgrade your EFK stack:

  1. Review how to specify logging Ansible variables and update your Ansible inventory file to at least set the following required variable in the [OSEv3:vars] section:

    [OSEv3:vars]

openshift_logging_install_logging=true (1)
    1 Enables the ability to upgrade the logging stack.

  2. Update any other openshift_logging_* variables that you want to override the default values for, as described in Specifying Logging Ansible Variables.

    You can set the openshift_logging_elasticsearch_replace_configmap parameter to true to replace your logging-elasticsearch ConfigMap with the current default values. In some cases, using an older ConfigMap can cause the upgrade to fail. The default is set to false. For more information, see the parameter in specify logging Ansible variables.

    If your Fluentd DeploymentConfig object and DaemonSet object for the EFK components are already set with:

            image: <image_name>:<vX.Y>
        imagePullPolicy: IfNotPresent

    The latest version <image_name> might not be pulled if there is already one with the same <image_name:vX.Y> stored locally on the node where the pod is being re-deployed. If your image version is v3.11, and you want to upgrade to the latest version using the playbook, set the openshift_image_tag=v3.11.<Z> or oreg_url=registry.access.redhat.com/openshift3/ose-${component}:v3.11.<Z> Ansible parameter.

  3. Dechedule your Fluentd pods to stop data ingestion and ensure the cluster state does not change.

    For example, you can change the node selector in Fluentd pods to one that does not match any nodes.

    oc patch daemonset logging-fluentd -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'

  4. Perform an Elasticsearch Index flush on all relevant indices. The flush process persists all logs from memory to disk, which prevents log loss when Elasticsearch is shutdown during the upgrade.

  5. Perform an online or offline backup:

    • Perform an online backup of specific Elasticsearch indices the entire cluster.

    • Perform an offline backup:

      1. Scale down all Elasticsearch DeploymentConfig to 0:

        $ oc scale dc <name> -n openshift-logging --replicas=0

      2. Back up external persistent volumes using the appropriate method for your organization.

  6. For any file name with a dot character, you need to take one of the following actions before upgrading:

    • Deleting the indices. This is the better approach to avoid mapping conflicts during the upgrade.

    • Reindexing the data and changing the documents to get rid of conflicting data structure. This method retain the data. For information on potential mapping conflicts see Mappging changes in the Elasticsearch documentation.

  7. Repeat the on-line or offline backup.

  8. Run the openshift-logging/config.yml playbook according to the deploying the EFK stack instructions to complete the logging upgrade. You run the installation playbook for the new OpenShift Container Platform version to upgrade the logging deployment.

Upgrading if fields do not have dots

If the script above indicates your indices do not contain fields with a dot in the name, use the following steps to upgrade.

  1. Review how to specify logging Ansible variables and update your Ansible inventory file to at least set the following required variable in the [OSEv3:vars] section:

    [OSEv3:vars]

openshift_logging_install_logging=true (1)
    1 Enables the ability to upgrade the logging stack.

  2. Update any other openshift_logging_* variables that you want to override the default values for, as described in Specifying Logging Ansible Variables.

    You can set the openshift_logging_elasticsearch_replace_configmap parameter to true to replace your logging-elasticsearch ConfigMap with the current default values. In some cases, using an older ConfigMap can cause the upgrade to fail. The default is set to false. For more information, see the parameter in specify logging Ansible variables.

    If your Fluentd DeploymentConfig object and DaemonSet object for the EFK components are already set with:

            image: <image_name>:<vX.Y>
        imagePullPolicy: IfNotPresent

    The latest version <image_name> might not be pulled if there is already one with the same <image_name:vX.Y> stored locally on the node where the pod is being re-deployed. If your image version is v3.11, and you want to upgrade to the latest version using the playbook, set the openshift_image_tag=v3.11.<Z> or oreg_url=registry.access.redhat.com/openshift3/ose-${component}:v3.11.<Z> Ansible parameter.

  3. Optionally, dechedule your Fluentd pods and scale down your Elasticsearch pods to stop data ingestion and ensure the cluster state does not change.

    For example, you can change the node selector in Fluentd pods to one that does not match any nodes.

    oc patch daemonset logging-fluentd -p '{"spec": {"template": {"spec": {"nodeSelector": {"non-existing": "true"}}}}}'

  4. Optionally, perform and online or offline backup:

    • Perform an online backup of specific Elasticsearch indices the entire cluster.

    • Perform an offline backup:

      1. Scale down all Elasticsearch DeploymentConfigs to 0:

        $ oc scale dc <name> -n openshift-logging --replicas=0

      2. Back up external persistent volumes using the appropriate method for your organization.

  5. Run the openshift-logging/config.yml playbook according to the deploying the EFK stack instructions to complete the logging upgrade. You run the installation playbook for the new OpenShift Container Platform version to upgrade the logging deployment.

  6. Optionally, use the Elasticsearch restore module to restore your Elasticsearch indices from the snapshot.

Upgrading cluster metrics

To upgrade an existing cluster metrics deployment, you review your parameters and run the openshift-metrics/config.yml playbook.

  1. Review how to specify metrics Ansible variables and update your Ansible inventory file to at least set the following required variable in the [OSEv3:vars] section:

    [OSEv3:vars]

openshift_metrics_install_metrics=true (1)
openshift_metrics_hawkular_hostname=<fqdn> (2)
openshift_metrics_cassandra_storage_type=(emptydir|pv|dynamic) (3)
    1 Enables the ability to upgrade the metrics deployment.
    2 Used for the Hawkular Metrics route. Specify a fully qualified domain name.
    3 Specify the same type as the previous deployment.

  2. Update any other openshift_metrics_* variables that you want to override the default values for, as described in Specifying Metrics Ansible Variables.

  3. Run the openshift-metrics/config.yml playbook according to the deploying the metrics deployment instructions to complete the metrics upgrade. You run the installation playbook for the new OpenShift Container Platform version to upgrade the logging deployment.

Verifying the upgrade

Ensure that:

  • The cluster is healthy.

  • The master, node, and etcd services or static pods are running well.

  • The OpenShift Container Platform, docker-registry, and router versions are correct.

  • The original applications are still available, and new application can be created.

  • Running oc adm diagnostics produces no errors.

To verify the upgrade:

  1. Check that all nodes are marked as Ready:

    # oc get nodes
NAME                     STATUS    ROLES        AGE       VERSION
master1.example.com      Ready     master       47d       v1.11.0+d4cacc0
master2.example.com      Ready     master       47d       v1.11.0+d4cacc0
master3.example.com      Ready     master       47d       v1.11.0+d4cacc0
infra-node1.example.com  Ready     infra        47d       v1.11.0+d4cacc0
infra-node2.example.com  Ready     infra        47d       v1.11.0+d4cacc0
node1.example.com        Ready     compute      47d       v1.11.0+d4cacc0
node2.example.com        Ready     compute      47d       v1.11.0+d4cacc0

  2. Verify that the static pods for the control plane are running:

    # oc get pods -n kube-system
NAME                                 READY     STATUS    RESTARTS   AGE
master-api-master1.example.com           1/1       Running   4          1h
master-controllers-master1.example.com   1/1       Running   3          1h
master-etcd-master1.example.com          1/1       Running   6          5d
[...]

  3. Verify that you are running the expected versions of the docker-registry and router images, if deployed:

    # oc get -n default dc/docker-registry -o json | grep \"image\"
    "image": "openshift3/ose-docker-registry:v3.11.634",
# oc get -n default dc/router -o json | grep \"image\"
    "image": "openshift3/ose-haproxy-router:v3.11.634",

  4. Use the diagnostics tool on the master to look for common issues:

    # oc adm diagnostics
...
[Note] Summary of diagnostics execution:
[Note] Completed with no errors or warnings seen.