After updating the OpenShift Container Platform cluster from 4.4 to 4.5, you can then update the Elasticsearch Operator and Cluster Logging Operator from 4.4 to 4.5.

Cluster logging 4.5 introduces a new Elasticsearch version, Elasticsearch 6.8.1, and an enhanced security plug-in, Open Distro for Elasticsearch. The new Elasticsearch version introduces a new Elasticsearch data model, where the Elasticsearch data is indexed only by type: infrastructure, application, and audit. Previously, data was indexed by type (infrastructure and application) and project.

Because of the new data model, the update does not migrate existing custom Kibana index patterns and visualizations into the new version. You must re-create your Kibana index patterns and visualizations to match the new indices after updating.

Due to the nature of these changes, you are not required to update your cluster logging to 4.5. However, when you update to OpenShift Container Platform 4.6, you must update cluster logging to 4.6 at that time.

Updating cluster logging

After updating the OpenShift Container Platform cluster, you can update cluster logging from 4.4 to 4.5 by changing the subscription for the Elasticsearch Operator and the Cluster Logging Operator.

When you update:

  • You must update the Elasticsearch Operator before updating the Cluster Logging Operator.

  • You must update both the Elasticsearch Operator and the Cluster Logging Operator.

    Kibana is unusable when the Elasticsearch Operator has been updated but the Cluster Logging Operator has not been updated.

    If you update the Cluster Logging Operator before the Elasticsearch Operator, Kibana does not update and the Kibana Custom Resource (CR) is not created. To work around this problem, delete the Cluster Logging Operator Pod. When the Cluster Logging Operator Pod redeploys, the Kibana CR is created.

If your cluster logging version is prior to 4.4, you must upgrade cluster logging to 4.4 before updating to 4.5.

Prerequisites
  • Update the OpenShift Container Platform cluster from 4.4 to 4.5.

  • Make sure the cluster logging status is healthy:

    • All Pods are ready.

    • The Elasticsearch cluster is healthy.

  • Back up your Elasticsearch and Kibana data.

Procedure
  1. Update the Elasticsearch Operator:

    1. From the web console, click OperatorsInstalled Operators.

    2. Select the openshift-operators-redhat project.

    3. Click the Elasticsearch Operator.

    4. Click SubscriptionChannel.

    5. In the Change Subscription Update Channel window, select 4.5 and click Save.

    6. Wait for a few seconds, then click OperatorsInstalled Operators.

      The Elasticsearch Operator is shown as 4.5. For example:

      Elasticsearch Operator
      4.5.0-202007012112.p0 provided
      by Red Hat, Inc

      Wait for the Status field to report Succeeded.

  2. Update the Cluster Logging Operator:

    1. From the web console, click OperatorsInstalled Operators.

    2. Select the openshift-logging project.

    3. Click the Cluster Logging Operator.

    4. Click SubscriptionChannel.

    5. In the Change Subscription Update Channel window, select 4.5 and click Save.

    6. Wait for a few seconds, then click OperatorsInstalled Operators.

      The Cluster Logging Operator is shown as 4.5. For example:

      Cluster Logging
      4.5.0-202007012112.p0 provided
      by Red Hat, Inc

      Wait for the Status field to report Succeeded.

  3. Check the logging components:

    1. Ensure that all Elasticsearch Pods are in the Ready status:

      $ oc get pod -n openshift-logging --selector component=elasticsearch
      
      NAME                                            READY   STATUS    RESTARTS   AGE
      elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk   2/2     Running   0          31m
      elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk   2/2     Running   0          30m
      elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc     2/2     Running   0          29m
    2. Ensure that the Elasticsearch cluster is healthy:

      $ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- es_cluster_health
      {
        "cluster_name" : "elasticsearch",
        "status" : "green",
      }
      ....
    3. Ensure that the Elasticsearch CronJobs are created:

      $ oc project openshift-logging
      $ oc get CronJob elasticsearch-*
      NAME                           SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
      elasticsearch-delete-app       */15 * * * *   False     0        <none>          27s
      elasticsearch-delete-audit     */15 * * * *   False     0        <none>          27s
      elasticsearch-delete-infra     */15 * * * *   False     0        <none>          27s
      elasticsearch-rollover-app     */15 * * * *   False     0        <none>          27s
      elasticsearch-rollover-audit   */15 * * * *   False     0        <none>          27s
      elasticsearch-rollover-infra   */15 * * * *   False     0        <none>          27s
    4. Verify that the log store is updated to 4.5 and the indices are green:

      $ oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indices

      You should see the app-0000x, infra-0000x, audit-0000x, .security indices.

      Tue Jun 30 14:30:54 UTC 2020
      health status index                                                                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
      green  open   infra-000008                                                          bnBvUFEXTWi92z3zWAzieQ   3 1       222195            0        289            144
      green  open   infra-000004                                                          rtDSzoqsSl6saisSK7Au1Q   3 1       226717            0        297            148
      green  open   infra-000012                                                          RSf_kUwDSR2xEuKRZMPqZQ   3 1       227623            0        295            147
      green  open   .kibana_7                                                             1SJdCqlZTPWlIAaOUd78yg   1 1            4            0          0              0
      green  open   .operations.2020.06.30                                                aOHMYOa3S_69NJFh2t3yrQ   3 1      4206118            0       8998           4499
      green  open   project.local-storage.d5c8a3d6-30a3-4512-96df-67c537540072.2020.06.30 O_Uldg2wS5K_L6FyqWxOZg   3 1        91052            0        135             67
      green  open   infra-000010                                                          iXwL3bnqTuGEABbUDa6OVw   3 1       248368            0        317            158
      green  open   .searchguard                                                          rQhAbWuLQ9iuTsZeHi_2ew   1 1            5           64          0              0
      green  open   infra-000009                                                          YN9EsULWSNaxWeeNvOs0RA   3 1       258799            0        337            168
      green  open   infra-000014                                                          YP0U6R7FQ_GVQVQZ6Yh9Ig   3 1       223788            0        292            146
      green  open   infra-000015                                                          JRBbAbEmSMqK5X40df9HbQ   3 1       224371            0        291            145
      green  open   .orphaned.2020.06.30                                                  n_xQC2dWQzConkvQqei3YA   3 1            9            0          0              0
      green  open   infra-000007                                                          llkkAVSzSOmosWTSAJM_hg   3 1       228584            0        296            148
      green  open   infra-000005                                                          d9BoGQdiQASsS3BBFm2iRA   3 1       227987            0        297            148
      green  open   .kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac                      l911Z8dSI23py6GDtyJrA    1 1            5            4          0              0
      green  open   project.ui.29cb9680-864d-43b2-a6cf-134c837d6f0c.2020.06.30            5A_YdRlAT3m1Z-vbqBuGWA   3 1           24            0          0              0
      green  open   infra-000003                                                          1-goREK1QUKlQPAIVkWVaQ   3 1       226719            0        295            147
      green  open   .security                                                             zeT65uOuRTKZMjg_bbUc1g   1 1            5            0          0              0
      green  open   .kibana-377444158_kubeadmin                                           wvMhDwJkR-mRZQO84K0gUQ   3 1            1            0          0              0
      green  open   infra-000006                                                          5H-KBSXGQKiO7hdapDE23g   3 1       226676            0        295            147
      green  open   project.nw.6233ad57-aff0-4d5a-976f-370636f47b11.2020.06.30            dtc6J-nLSCC59EygeV41RQ   3 1           10            0          0              0
      green  open   infra-000001                                                          eH53BQ-bSxSWR5xYZB6lVg   3 1       341800            0        443            220
      green  open   .kibana-6                                                             RVp7TemSSemGJcsSUmuf3A   1 1            4            0          0              0
      green  open   infra-000011                                                          J7XWBauWSTe0jnzX02fU6A   3 1       226100            0        293            146
      green  open   app-000001                                                            axSAFfONQDmKwatkjPXdtw   3 1       103186            0        126             57
      green  open   infra-000016                                                          m9c1iRLtStWSF1GopaRyCg   3 1        13685            0         19              9
      green  open   infra-000002                                                          Hz6WvINtTvKcQzw-ewmbYg   3 1       228994            0        296            148
      green  open   project.qt.2c05acbd-bc12-4275-91ab-84d180b53505.2020.06.30            MUm3eFJjSPKQOJWoHskKqw   3 1        12262            0         14              7
      green  open   infra-000013                                                          KR9mMFUpQl-jraYtanyIGw   3 1       228166            0        298            148
      green  open   audit-000001                                                          eERqLdLmQOiQDFES1LBATQ   3 1            0            0          0              0
    5. Verify that the log collector is updated to 4.5:

      $ oc get ds fluentd -o json | grep fluentd-init

      You should see a fluentd-init container:

      "containerName": "fluentd-init"
    6. Verify that the log visualizer is updated to 4.5 using the Kibana CRD:

      $ oc get kibana kibana -o json

      You should see a Kibana Pod with the ready status:

      [
      {
      "clusterCondition": {
      "kibana-5fdd766ffd-nb2jj": [
      {
      "lastTransitionTime": "2020-06-30T14:11:07Z",
      "reason": "ContainerCreating",
      "status": "True",
      "type": ""
      },
      {
      "lastTransitionTime": "2020-06-30T14:11:07Z",
      "reason": "ContainerCreating",
      "status": "True",
      "type": ""
      }
      ]
      },
      "deployment": "kibana",
      "pods": {
      "failed": [],
      "notReady": []
      "ready": []
      },
      "replicaSets": [
      "kibana-5fdd766ffd"
      ],
      "replicas": 1
      }
      ]
    7. Verify the Curator is updated to 4.5:

      $ oc get cronjob -o name
      cronjob.batch/curator
      cronjob.batch/elasticsearch-delete-app
      cronjob.batch/elasticsearch-delete-audit
      cronjob.batch/elasticsearch-delete-infra
      cronjob.batch/elasticsearch-rollover-app
      cronjob.batch/elasticsearch-rollover-audit
      cronjob.batch/elasticsearch-rollover-infra

      You should see the elasticsearch-delete-* and elasticsearch-rollover-* indices.

Post update

If you use Kibana, after the Elasticsearch Operator and Cluster Logging Operator are fully updated to 4.5, you must perform the following actions to complete the upgrade:

  • Recreate your Kibana index patterns. Because of changes in the security plug-in, the cluster logging upgrade does not automatically create index patterns.

    • Regular users must manually create index patterns to see logs for their projects. Because of the new Elasticsearch data model, all logs that were stored in an index prefixed with project- in OpenShift Container Platform 4.4 are now in a set of indices prefixed with app-. Users should create a new index patterns named app and use the @timestamp time field to view their container logs. To view historical data gathered by cluster logging 4.4, users need to create index patterns using project* field.

    • Admin users need to create index patterns using the app, infra and audit prefix with the @timestamp time field. The infra- indices were previously .operations-. The audit- indices are the audit logs. To view historical data gathered by cluster logging 4.4, admin users need to create index patterns using the .operation* field to view system logs and create index patterns using the project* field to view project-level logs.

  • Recreate your Kibana Visualizations to use the new index patterns.