Update the OpenShift Dedicated cluster from 4.4 to 4.6.
Ensure the cluster logging status is healthy:
All pods are ready.
The Elasticsearch cluster is healthy.
Back up your Elasticsearch and Kibana data.
When you update the OpenShift Dedicated cluster, you can update cluster logging from 4.4 to 4.6 by changing the subscription for the Cluster Logging Operator and the Elasticsearch Operator.
When your cluster logging version is 4.4, you must update cluster logging as follows due to changes in the way that OLM provides operator packages.
Additionally, it is recommended that you update Cluster Logging Operator before you update Elasticsearch Operator to avoid the need to restart each Elasticsearch pod manually, one by one.
Perform each update, version by version, in the following order:
Update OpenShift Cluster to 4.5.
Update Cluster Logging Operator to the latest version of 4.4, then update to 4.5.
Update Elasticsearch Operator to the latest version of 4.4, then update to 4.5.
Update OpenShift Cluster to 4.6.
Update Cluster Logging Operator to latest version of 4.5, then update to 4.6.
Update Elasticsearch Operator to latest version of 4.5, then update to 4.6
|
If you have already updated to Openshift 4.6 with the cluster logging still on version 4.4, be aware that there is not any cluster logging 4.5 metadata in the
|
Update the Cluster Logging Operator:
From the OpenShift web console, click Operators → Installed Operators.
Select the openshift-logging project.
Click Cluster Logging.
Click Subscription → Channel.
In the Change Subscription Update Channel window, select the version that is one version newer than your current version.
On the YAML tab, set the retention policy in the clusterlogging custom resource (CR) as follows. You must specify the length of time that Elasticsearch should retain each log source. Enter an integer for the number of days, for example, 7d for seven days. The application maxAge can be less than or equal to 7d. The infra maxAge and audit maxAge must be 1h, and you must explicitly set this value to override the default. Logs older than the maxAge are deleted. You must specify a retention policy for each log source or the Elasticsearch indices will not be created for that source.
kind: "ClusterLogging"
metadata:
name: "instance"
namespace: "openshift-logging"
spec:
managementState: "Managed"
logStore:
type: "elasticsearch"
retentionPolicy:
application:
maxAge: 7d
infra:
maxAge: 1h
audit:
maxAge: 1h
...Click Save.
Wait for a few seconds, then click Operators → Installed Operators.
The Cluster Logging Operator is shown as 4.6. For example:
Cluster Logging
4.6.0-202007012112.p0 provided
by Red Hat, IncWait for the Status field to report Succeeded.
Update the Elasticsearch Operator:
From the OpenShift web console, click Operators → Installed Operators.
Select the openshift-operators-redhat project.
If the Elasticsearch Operator is installed on the openshift-logging namespace, uninstall the Elasticsearch Operator.
|
This uninstall action does not delete the existing data, volumes, and so on. |
Click Operators → OperatorHub. Search for and select OpenShift Elasticsearch Operator.
Click Install + The Install operator page opens.
For Update Channel, select the version that is one version newer than your current version.
For Installed Namespace, select openshift-operators-redhat namespace.
Click Install.
Wait for a few seconds, then click Operators → Installed Operators.
The Elasticsearch Operator is shown as 4.6. For example:
Elasticsearch Operator
4.6.0-202007012112.p0 provided
by Red Hat, IncWait for the Status field to report Succeeded.
Check the logging components:
Ensure that all Elasticsearch pods are in the Ready status:
$ oc get pod -n openshift-logging --selector component=elasticsearchNAME READY STATUS RESTARTS AGE
elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk 2/2 Running 0 31m
elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk 2/2 Running 0 30m
elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc 2/2 Running 0 29mEnsure that the Elasticsearch cluster is healthy:
$ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- es_cluster_health{
"cluster_name" : "elasticsearch",
"status" : "green",
}
....
Ensure that the Elasticsearch cron jobs are created:
$ oc project openshift-logging$ oc get cron jobNAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
curator */10 * * * * False 0 109s
elasticsearch-im-app */15 * * * * False 1 19s 107s
elasticsearch-im-audit */15 * * * * False 1 19s 107s
elasticsearch-im-infra */15 * * * * False 1 19s 107sVerify that the log store is updated to 4.6 and the indices are green:
$ oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indicesYou should see the app-0000x, infra-0000x, audit-0000x, .security indices.
|
It takes approximately 15 minutes for the indices to be updated. |
Tue Jun 30 14:30:54 UTC 2020
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open infra-000008 bnBvUFEXTWi92z3zWAzieQ 3 1 222195 0 289 144
green open infra-000004 rtDSzoqsSl6saisSK7Au1Q 3 1 226717 0 297 148
green open infra-000012 RSf_kUwDSR2xEuKRZMPqZQ 3 1 227623 0 295 147
green open .kibana_7 1SJdCqlZTPWlIAaOUd78yg 1 1 4 0 0 0
green open .operations.2020.06.30 aOHMYOa3S_69NJFh2t3yrQ 3 1 4206118 0 8998 4499
green open project.local-storage.d5c8a3d6-30a3-4512-96df-67c537540072.2020.06.30 O_Uldg2wS5K_L6FyqWxOZg 3 1 91052 0 135 67
green open infra-000010 iXwL3bnqTuGEABbUDa6OVw 3 1 248368 0 317 158
green open .searchguard rQhAbWuLQ9iuTsZeHi_2ew 1 1 5 64 0 0
green open infra-000009 YN9EsULWSNaxWeeNvOs0RA 3 1 258799 0 337 168
green open infra-000014 YP0U6R7FQ_GVQVQZ6Yh9Ig 3 1 223788 0 292 146
green open infra-000015 JRBbAbEmSMqK5X40df9HbQ 3 1 224371 0 291 145
green open .orphaned.2020.06.30 n_xQC2dWQzConkvQqei3YA 3 1 9 0 0 0
green open infra-000007 llkkAVSzSOmosWTSAJM_hg 3 1 228584 0 296 148
green open infra-000005 d9BoGQdiQASsS3BBFm2iRA 3 1 227987 0 297 148
green open .kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac l911Z8dSI23py6GDtyJrA 1 1 5 4 0 0
green open project.ui.29cb9680-864d-43b2-a6cf-134c837d6f0c.2020.06.30 5A_YdRlAT3m1Z-vbqBuGWA 3 1 24 0 0 0
green open infra-000003 1-goREK1QUKlQPAIVkWVaQ 3 1 226719 0 295 147
green open .security zeT65uOuRTKZMjg_bbUc1g 1 1 5 0 0 0
green open .kibana-377444158_kubeadmin wvMhDwJkR-mRZQO84K0gUQ 3 1 1 0 0 0
green open infra-000006 5H-KBSXGQKiO7hdapDE23g 3 1 226676 0 295 147
green open project.nw.6233ad57-aff0-4d5a-976f-370636f47b11.2020.06.30 dtc6J-nLSCC59EygeV41RQ 3 1 10 0 0 0
green open infra-000001 eH53BQ-bSxSWR5xYZB6lVg 3 1 341800 0 443 220
green open .kibana-6 RVp7TemSSemGJcsSUmuf3A 1 1 4 0 0 0
green open infra-000011 J7XWBauWSTe0jnzX02fU6A 3 1 226100 0 293 146
green open app-000001 axSAFfONQDmKwatkjPXdtw 3 1 103186 0 126 57
green open infra-000016 m9c1iRLtStWSF1GopaRyCg 3 1 13685 0 19 9
green open infra-000002 Hz6WvINtTvKcQzw-ewmbYg 3 1 228994 0 296 148
green open project.qt.2c05acbd-bc12-4275-91ab-84d180b53505.2020.06.30 MUm3eFJjSPKQOJWoHskKqw 3 1 12262 0 14 7
green open infra-000013 KR9mMFUpQl-jraYtanyIGw 3 1 228166 0 298 148
green open audit-000001 eERqLdLmQOiQDFES1LBATQ 3 1 0 0 0 0Verify that the log collector is updated to 4.6:
$ oc get ds fluentd -o json -- output=jsonpath='{.spec.template.spec.initContainers[*].name}'You should see a fluentd-init container.
fluentd-initVerify that the log visualizer is updated to 4.6 using the Kibana CRD:
$ oc get kibana kibana -o jsonYou should see a Kibana pod with the ready status:
[
{
"clusterCondition": {
"kibana-5fdd766ffd-nb2jj": [
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
},
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
}
]
},
"deployment": "kibana",
"pods": {
"failed": [],
"notReady": []
"ready": []
},
"replicaSets": [
"kibana-5fdd766ffd"
],
"replicas": 1
}
]Verify the Curator is updated to 4.6:
$ oc get cronjob -o namecronjob.batch/curator
cronjob.batch/elasticsearch-im-app
cronjob.batch/elasticsearch-im-audit
cronjob.batch/elasticsearch-im-infraYou should see the elasticsearch-delete-* and elasticsearch-rollover-* cronjobs approximately 30 minutes after an installation or update.
If you use the Log Forwarding API to forward logs, after the Elasticsearch Operator and Cluster Logging Operator are fully updated to 4.6, you must replace your LogForwarding custom resource (CR) with a ClusterLogForwarder CR.
After updating the OpenShift Dedicated cluster, you can update cluster logging from 4.5 to 4.6 by changing the subscription for the Elasticsearch Operator and the Cluster Logging Operator.
Updating from 4.5:
You must update the Elasticsearch Operator before updating the Cluster Logging Operator.
You must update both the Elasticsearch Operator and the Cluster Logging Operator.
Kibana is unusable when the Elasticsearch Operator has been updated but the Cluster Logging Operator has not been updated.
If you update the Cluster Logging Operator before the Elasticsearch Operator, Kibana does not update and the Kibana custom resource (CR) is not created. To work around this problem, delete the Cluster Logging Operator pod. When the Cluster Logging Operator pod redeploys, the Kibana CR is created.
Update the Elasticsearch Operator:
From the OpenShift web console, click Operators → Installed Operators.
Select the openshift-operators-redhat project.
If the Elasticsearch Operator is installed on the openshift-logging namespace, uninstall the Elasticsearch Operator.
|
This uninstall action does not delete the existing data, volumes, and so on. |
Click Operators → OperatorHub. Search for and select OpenShift Elasticsearch Operator.
Click Install + The Install operator page opens.
For Update Channel, make sure 4.6 is selected.
For Installed Namepsace, select openshift-operators-redhat namespace.
Click Install.
Wait for a few seconds, then click Operators → Installed Operators.
The Elasticsearch Operator is shown as 4.6. For example:
Elasticsearch Operator
4.6.0-202007012112.p0 provided
by Red Hat, IncWait for the Status field to report Succeeded.
Update the Cluster Logging Operator:
From the OpenShift web console, click Operators → Installed Operators.
Select the openshift-logging project.
Click Cluster Logging.
Click Subscription → Channel.
In the Change Subscription Update Channel window, select 4.6.
On the YAML tab, set the retention policy in the clusterlogging custom resource (CR) as follows. You must specify the length of time that Elasticsearch should retain each log source. Enter an integer for the number of days, for example, 7d for seven days. The application maxAge can be less than or equal to 7d. The infra maxAge and audit maxAge must be 1h, and you must explicitly set this value to override the default. Logs older than the maxAge are deleted. You must specify a retention policy for each log source or the Elasticsearch indices will not be created for that source.
kind: "ClusterLogging"
metadata:
name: "instance"
namespace: "openshift-logging"
spec:
managementState: "Managed"
logStore:
type: "elasticsearch"
retentionPolicy:
application:
maxAge: 7d
infra:
maxAge: 1h
audit:
maxAge: 1h
...Click Save.
Wait for a few seconds, then click Operators → Installed Operators.
The Cluster Logging Operator is shown as 4.6. For example:
Cluster Logging
4.6.0-202007012112.p0 provided
by Red Hat, IncWait for the Status field to report Succeeded.
Check the logging components:
Ensure that all Elasticsearch pods are in the Ready status:
$ oc get pod -n openshift-logging --selector component=elasticsearchNAME READY STATUS RESTARTS AGE
elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk 2/2 Running 0 31m
elasticsearch-cdm-1pbrl44l-2-5c6d87589f-gx5hk 2/2 Running 0 30m
elasticsearch-cdm-1pbrl44l-3-88df5d47-m45jc 2/2 Running 0 29mEnsure that the Elasticsearch cluster is healthy:
$ oc exec -n openshift-logging -c elasticsearch elasticsearch-cdm-1pbrl44l-1-55b7546f4c-mshhk -- es_cluster_health{
"cluster_name" : "elasticsearch",
"status" : "green",
}
....
Ensure that the Elasticsearch cron jobs are created:
$ oc project openshift-logging$ oc get cron jobNAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
curator */10 * * * * False 0 109s
elasticsearch-im-app */15 * * * * False 1 19s 107s
elasticsearch-im-audit */15 * * * * False 1 19s 107s
elasticsearch-im-infra */15 * * * * False 1 19s 107sVerify that the log store is updated to 4.6 and the indices are green:
$ oc exec -c elasticsearch <any_es_pod_in_the_cluster> -- indicesYou should see the app-0000x, infra-0000x, audit-0000x, .security indices.
|
It takes approximately 15 minutes for the indices to be updated. |
Tue Jun 30 14:30:54 UTC 2020
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open infra-000008 bnBvUFEXTWi92z3zWAzieQ 3 1 222195 0 289 144
green open infra-000004 rtDSzoqsSl6saisSK7Au1Q 3 1 226717 0 297 148
green open infra-000012 RSf_kUwDSR2xEuKRZMPqZQ 3 1 227623 0 295 147
green open .kibana_7 1SJdCqlZTPWlIAaOUd78yg 1 1 4 0 0 0
green open .operations.2020.06.30 aOHMYOa3S_69NJFh2t3yrQ 3 1 4206118 0 8998 4499
green open project.local-storage.d5c8a3d6-30a3-4512-96df-67c537540072.2020.06.30 O_Uldg2wS5K_L6FyqWxOZg 3 1 91052 0 135 67
green open infra-000010 iXwL3bnqTuGEABbUDa6OVw 3 1 248368 0 317 158
green open .searchguard rQhAbWuLQ9iuTsZeHi_2ew 1 1 5 64 0 0
green open infra-000009 YN9EsULWSNaxWeeNvOs0RA 3 1 258799 0 337 168
green open infra-000014 YP0U6R7FQ_GVQVQZ6Yh9Ig 3 1 223788 0 292 146
green open infra-000015 JRBbAbEmSMqK5X40df9HbQ 3 1 224371 0 291 145
green open .orphaned.2020.06.30 n_xQC2dWQzConkvQqei3YA 3 1 9 0 0 0
green open infra-000007 llkkAVSzSOmosWTSAJM_hg 3 1 228584 0 296 148
green open infra-000005 d9BoGQdiQASsS3BBFm2iRA 3 1 227987 0 297 148
green open .kibana.647a750f1787408bf50088234ec0edd5a6a9b2ac l911Z8dSI23py6GDtyJrA 1 1 5 4 0 0
green open project.ui.29cb9680-864d-43b2-a6cf-134c837d6f0c.2020.06.30 5A_YdRlAT3m1Z-vbqBuGWA 3 1 24 0 0 0
green open infra-000003 1-goREK1QUKlQPAIVkWVaQ 3 1 226719 0 295 147
green open .security zeT65uOuRTKZMjg_bbUc1g 1 1 5 0 0 0
green open .kibana-377444158_kubeadmin wvMhDwJkR-mRZQO84K0gUQ 3 1 1 0 0 0
green open infra-000006 5H-KBSXGQKiO7hdapDE23g 3 1 226676 0 295 147
green open project.nw.6233ad57-aff0-4d5a-976f-370636f47b11.2020.06.30 dtc6J-nLSCC59EygeV41RQ 3 1 10 0 0 0
green open infra-000001 eH53BQ-bSxSWR5xYZB6lVg 3 1 341800 0 443 220
green open .kibana-6 RVp7TemSSemGJcsSUmuf3A 1 1 4 0 0 0
green open infra-000011 J7XWBauWSTe0jnzX02fU6A 3 1 226100 0 293 146
green open app-000001 axSAFfONQDmKwatkjPXdtw 3 1 103186 0 126 57
green open infra-000016 m9c1iRLtStWSF1GopaRyCg 3 1 13685 0 19 9
green open infra-000002 Hz6WvINtTvKcQzw-ewmbYg 3 1 228994 0 296 148
green open project.qt.2c05acbd-bc12-4275-91ab-84d180b53505.2020.06.30 MUm3eFJjSPKQOJWoHskKqw 3 1 12262 0 14 7
green open infra-000013 KR9mMFUpQl-jraYtanyIGw 3 1 228166 0 298 148
green open audit-000001 eERqLdLmQOiQDFES1LBATQ 3 1 0 0 0 0Verify that the log collector is updated to 4.6:
$ oc get ds fluentd -o json -- output=jsonpath='{.spec.template.spec.initContainers[*].name}'You should see a fluentd-init container.
fluentd-initVerify that the log visualizer is updated to 4.6 using the Kibana CRD:
$ oc get kibana kibana -o jsonYou should see a Kibana pod with the ready status:
[
{
"clusterCondition": {
"kibana-5fdd766ffd-nb2jj": [
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
},
{
"lastTransitionTime": "2020-06-30T14:11:07Z",
"reason": "ContainerCreating",
"status": "True",
"type": ""
}
]
},
"deployment": "kibana",
"pods": {
"failed": [],
"notReady": []
"ready": []
},
"replicaSets": [
"kibana-5fdd766ffd"
],
"replicas": 1
}
]Verify the Curator is updated to 4.6:
$ oc get cronjob -o namecronjob.batch/curator
cronjob.batch/elasticsearch-im-app
cronjob.batch/elasticsearch-im-audit
cronjob.batch/elasticsearch-im-infraYou should see the elasticsearch-delete-* and elasticsearch-rollover-* cronjobs approximately 30 minutes after an installation or update.
If you use the Log Forwarding API to forward logs, after the Elasticsearch Operator and Cluster Logging Operator are fully updated to 4.6, you must replace your LogForwarding custom resource (CR) with a ClusterLogForwarder CR.