You can install cluster logging by deploying the Elasticsearch and Cluster Logging Operators. The Elasticsearch Operator creates and manages the Elasticsearch cluster used by cluster logging. The Cluster Logging Operator creates and manages the components of the logging stack.

The process for deploying cluster logging to OpenShift Container Platform involves:

Installing cluster logging using the web console

You can use the OpenShift Container Platform web console to install the Elasticsearch and Cluster Logging operators.

Prerequisites
  • Ensure that you have the necessary persistent storage for Elasticsearch. Note that each Elasticsearch node requires its own storage volume.

    Elasticsearch is a memory-intensive application. By default, OpenShift Container Platform installs three Elasticsearch nodes with memory requests and limits of 16 GB. This initial set of three OpenShift Container Platform nodes might not have enough memory to run Elasticsearch within your cluster. If you experience memory issues that are related to Elasticsearch, you should add more Elasticsearch nodes to your cluster rather than increasing the memory on exiting nodes.

Procedure

To install the Elasticsearch Operator and Cluster Logging Operator using the OpenShift Container Platform web console:

  1. Install the Elasticsearch Operator:

    1. In the OpenShift Container Platform web console, click OperatorsOperatorHub.

    2. Choose Elasticsearch Operator from the list of available Operators, and click Install.

    3. Ensure that the All namespaces on the cluster is selected under Installation Mode.

    4. Ensure that openshift-operators-redhat is selected under Installed Namespace.

      You must specify the openshift-operators-redhat namespace. The openshift-operators namespace might contain Community Operators, which are untrusted and could publish a metric with the same name as an OpenShift Container Platform metric, which would cause conflicts.

    5. Select Enable operator recommended cluster monitoring on this namespace.

      This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-operators-redhat namespace.

    6. Select 4.5 as the Update Channel.

    7. Select an Approval Strategy.

      • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.

      • The Manual strategy requires a user with appropriate credentials to approve the Operator update.

    8. Click Install.

    9. Verify that the Elasticsearch Operator installed by switching to the OperatorsInstalled Operators page.

    10. Ensure that Elasticsearch Operator is listed in all projects with a Status of Succeeded.

  2. Install the Cluster Logging Operator:

    1. In the OpenShift Container Platform web console, click OperatorsOperatorHub.

    2. Choose Cluster Logging from the list of available Operators, and click Install.

    3. Ensure that the A specific namespace on the cluster is selected under Installation Mode.

    4. Ensure that Operator recommended namespace is openshift-logging under Installed Namespace.

    5. Select Enable operator recommended cluster monitoring on this namespace.

      This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-logging namespace.

    6. Select 4.5 as the Update Channel.

    7. Select an Approval Strategy.

      • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.

      • The Manual strategy requires a user with appropriate credentials to approve the Operator update.

    8. Click Install.

    9. Verify that the Cluster Logging Operator installed by switching to the OperatorsInstalled Operators page.

    10. Ensure that Cluster Logging is listed in the openshift-logging project with a Status of Succeeded.

      If the Operator does not appear as installed, to troubleshoot further:

      • Switch to the OperatorsInstalled Operators page and inspect the Status column for any errors or failures.

      • Switch to the WorkloadsPods page and check the logs in any Pods in the openshift-logging project that are reporting issues.

  3. Create a cluster logging instance:

    1. Switch to the AdministrationCustom Resource Definitions page.

    2. On the Custom Resource Definitions page, click ClusterLogging.

    3. On the Custom Resource Definition Overview page, select View Instances from the Actions menu.

    4. On the ClusterLoggings page, click Create ClusterLogging.

      You might have to refresh the page to load the data.

    5. In the YAML field, replace the code with the following:

      This default cluster logging configuration should support a wide array of environments. Review the topics on tuning and configuring the cluster logging components for information on modifications you can make to your cluster logging cluster.

      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance" (1)
        namespace: "openshift-logging"
      spec:
        managementState: "Managed"  (2)
        logStore:
          type: "elasticsearch"  (3)
          retentionPolicy: (4)
            application:
              maxAge: 1d
            infra:
              maxAge: 7d
            audit:
              maxAge: 7d
          elasticsearch:
            nodeCount: 3 (5)
            storage:
              storageClassName: "<storage-class-name>" (6)
              size: 200G
            redundancyPolicy: "SingleRedundancy"
        visualization:
          type: "kibana"  (7)
          kibana:
            replicas: 1
        curation:
          type: "curator"
          curator:
            schedule: "30 3 * * *" (8)
        collection:
          logs:
            type: "fluentd"  (9)
            fluentd: {}
      1 The name must be instance.
      2 The cluster logging management state. In some cases, if you change the cluster logging defaults, you must set this to Unmanaged. However, an unmanaged deployment does not receive updates until the cluster logging is placed back into a managed state.
      3 Settings for configuring Elasticsearch. Using the CR, you can configure shard replication policy and persistent storage.
      4 Specify the length of time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 7d for seven days. Logs older than the maxAge are deleted. You must specify a retention policy for each log source or the Elasticsearch indices will not be created for that source.
      5 Specify the number of Elasticsearch nodes. See the note that follows this list.
      6 Enter the name of an existing StorageClass for Elasticsearch storage. For best performance, specify a StorageClass that allocates block storage. If you do not specify a StorageClass, OpenShift Container Platform deploys cluster logging with ephemeral storage only.
      7 Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana nodes. For more information, see Configuring Kibana.
      8 Settings for configuring the Curator schedule. Curator is used to remove data that is in the Elasticsearch index format prior to OpenShift Container Platform 4.5 and will be removed in a later release.
      9 Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits. For more information, see Configuring Fluentd.

      The maximum number of Elasticsearch master nodes is three. If you specify a nodeCount greater than 3, OpenShift Container Platform creates three Elasticsearch nodes that are Master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as Data-only nodes, using client and data roles. Master nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded.

      For example, if nodeCount=4, the following nodes are created:

      $ oc get deployment
      Example output
      cluster-logging-operator       1/1     1            1           18h
      elasticsearch-cd-x6kdekli-1    0/1     1            0           6m54s
      elasticsearch-cdm-x6kdekli-1   1/1     1            1           18h
      elasticsearch-cdm-x6kdekli-2   0/1     1            0           6m49s
      elasticsearch-cdm-x6kdekli-3   0/1     1            0           6m44s

      The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes.

    6. Click Create. This creates the Cluster Logging components, the Elasticsearch Custom Resource and components, and the Kibana interface.

  4. Verify the install:

    1. Switch to the WorkloadsPods page.

    2. Select the openshift-logging project.

      You should see several Pods for cluster logging, Elasticsearch, Fluentd, and Kibana similar to the following list:

      • cluster-logging-operator-cb795f8dc-xkckc

      • elasticsearch-cdm-b3nqzchd-1-5c6797-67kfz

      • elasticsearch-cdm-b3nqzchd-2-6657f4-wtprv

      • elasticsearch-cdm-b3nqzchd-3-588c65-clg7g

      • fluentd-2c7dg

      • fluentd-9z7kk

      • fluentd-br7r2

      • fluentd-fn2sb

      • fluentd-pb2f8

      • fluentd-zqgqx

      • kibana-7fb4fd4cc9-bvt4p

Post-installation tasks

If you plan to use Kibana, you must manually create your Kibana index patterns and visualizations to explore and visualize data in Kibana.

Installing cluster logging using the CLI

You can use the OpenShift Container Platform CLI to install the Elasticsearch and Cluster Logging operators.

Prerequisites
  • Ensure that you have the necessary persistent storage for Elasticsearch. Note that each Elasticsearch node requires its own storage volume.

    Elasticsearch is a memory-intensive application. By default, OpenShift Container Platform installs three Elasticsearch nodes with memory requests and limits of 16 GB. This initial set of three OpenShift Container Platform nodes might not have enough memory to run Elasticsearch within your cluster. If you experience memory issues that are related to Elasticsearch, you should add more Elasticsearch nodes to your cluster rather than increasing the memory on exiting nodes.

Procedure

To install the Elasticsearch Operator and Cluster Logging Operator using the CLI:

  1. Create a Namespace for the Elasticsearch Operator.

    1. Create a Namespace object YAML file (for example, eo-namespace.yaml) for the Elasticsearch Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-operators-redhat (1)
        annotations:
          openshift.io/node-selector: ""
        labels:
          openshift.io/cluster-monitoring: "true" (2)
      1 You must specify the openshift-operators-redhat Namespace. To prevent possible conflicts with metrics, you should configure the Prometheus Cluster Monitoring stack to scrape metrics from the openshift-operators-redhat Namespace and not the openshift-operators Namespace. The openshift-operators Namespace might contain Community Operators, which are untrusted and could publish a metric with the same name as an OpenShift Container Platform metric, which would cause conflicts.
      2 You must specify this label as shown to ensure that cluster monitoring scrapes the openshift-operators-redhat Namespace.
    2. Create the Namespace:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f eo-namespace.yaml
  2. Create a Namespace for the Cluster Logging Operator:

    1. Create a Namespace object YAML file (for example, clo-namespace.yaml) for the Cluster Logging Operator:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-logging
        annotations:
          openshift.io/node-selector: ""
        labels:
          openshift.io/cluster-monitoring: "true"
    2. Create the Namespace:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f clo-namespace.yaml
  3. Install the Elasticsearch Operator by creating the following objects:

    1. Create an Operator Group object YAML file (for example, eo-og.yaml) for the Elasticsearch operator:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-operators-redhat
        namespace: openshift-operators-redhat (1)
      spec: {}
      1 You must specify the openshift-operators-redhat Namespace.
    2. Create an Operator Group object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f eo-og.yaml
    3. Create a Subscription object YAML file (for example, eo-sub.yaml) to subscribe a Namespace to the Elasticsearch Operator.

      Example Subscription
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: "elasticsearch-operator"
        namespace: "openshift-operators-redhat" (1)
      spec:
        channel: "4.5" (2)
        installPlanApproval: "Automatic"
        source: "redhat-operators" (3)
        sourceNamespace: "openshift-marketplace"
        name: "elasticsearch-operator"
      1 You must specify the openshift-operators-redhat Namespace.
      2 Specify 4.5 as the channel.
      3 Specify redhat-operators. If your OpenShift Container Platform cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object created when you configured the Operator Lifecycle Manager (OLM).
    4. Create the Subscription object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f eo-sub.yaml

      The Elasticsearch Operator is installed to the openshift-operators-redhat Namespace and copied to each project in the cluster.

    5. Verify the Operator installation:

      oc get csv --all-namespaces
      Example output
      NAMESPACE                                               NAME                                            DISPLAY                  VERSION               REPLACES   PHASE
      default                                                 elasticsearch-operator.4.5.0-202007012112.p0    Elasticsearch Operator   4.5.0-202007012112.p0               Succeeded
      kube-node-lease                                         elasticsearch-operator.4.5.0-202007012112.p0    Elasticsearch Operator   4.5.0-202007012112.p0               Succeeded
      kube-public                                             elasticsearch-operator.4.5.0-202007012112.p0    Elasticsearch Operator   4.5.0-202007012112.p0               Succeeded
      kube-system                                             elasticsearch-operator.4.5.0-202007012112.p0    Elasticsearch Operator   4.5.0-202007012112.p0               Succeeded
      openshift-apiserver-operator                            elasticsearch-operator.4.5.0-202007012112.p0    Elasticsearch Operator   4.5.0-202007012112.p0               Succeeded
      openshift-apiserver                                     elasticsearch-operator.4.5.0-202007012112.p0    Elasticsearch Operator   4.5.0-202007012112.p0               Succeeded
      openshift-authentication-operator                       elasticsearch-operator.4.5.0-202007012112.p0    Elasticsearch Operator   4.5.0-202007012112.p0               Succeeded
      openshift-authentication                                elasticsearch-operator.4.5.0-202007012112.p0    Elasticsearch Operator   4.5.0-202007012112.p0               Succeeded
      ...

      There should be an Elasticsearch Operator in each Namespace. The version number might be different than shown.

  4. Install the Cluster Logging Operator by creating the following objects:

    1. Create an OperatorGroup object YAML file (for example, clo-og.yaml) for the Cluster Logging Operator:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: cluster-logging
        namespace: openshift-logging (1)
      spec:
        targetNamespaces:
        - openshift-logging (1)
      1 You must specify the openshift-logging namespace.
    2. Create the OperatorGroup object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f clo-og.yaml
    3. Create a Subscription object YAML file (for example, clo-sub.yaml) to subscribe a Namespace to the Cluster Logging Operator.

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: cluster-logging
        namespace: openshift-logging (1)
      spec:
        channel: "4.5" (2)
        name: cluster-logging
        source: redhat-operators (3)
        sourceNamespace: openshift-marketplace
      1 You must specify the openshift-logging Namespace.
      2 Specify 4.5 as the channel.
      3 Specify redhat-operators. If your OpenShift Container Platform cluster is installed on a restricted network, also known as a disconnected cluster, specify the name of the CatalogSource object you created when you configured the Operator Lifecycle Manager (OLM).
    4. Create the Subscription object:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f clo-sub.yaml

      The Cluster Logging Operator is installed to the openshift-logging Namespace.

    5. Verify the Operator installation.

      There should be a Cluster Logging Operator in the openshift-logging Namespace. The Version number might be different than shown.

      oc get csv -n openshift-logging
      Example output
      NAMESPACE                                               NAME                                         DISPLAY                  VERSION               REPLACES   PHASE
      ...
      openshift-logging                                       clusterlogging.4.5.0-202007012112.p0         Cluster Logging          4.5.0-202007012112.p0              Succeeded
      ...
  5. Create a Cluster Logging instance:

    1. Create an instance object YAML file (for example, clo-instance.yaml) for the Cluster Logging Operator:

      This default Cluster Logging configuration should support a wide array of environments. Review the topics on tuning and configuring the Cluster Logging components for information on modifications you can make to your Cluster Logging cluster.

      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance" (1)
        namespace: "openshift-logging"
      spec:
        managementState: "Managed"  (2)
        logStore:
          type: "elasticsearch"  (3)
          retentionPolicy: (4)
            application:
              maxAge: 1d
            infra:
              maxAge: 7d
            audit:
              maxAge: 7d
          elasticsearch:
            nodeCount: 3 (5)
            storage:
              storageClassName: "<storage-class-name>" (6)
              size: 200G
            redundancyPolicy: "SingleRedundancy"
        visualization:
          type: "kibana"  (7)
          kibana:
            replicas: 1
        curation:
          type: "curator"
          curator:
            schedule: "30 3 * * *" (8)
        collection:
          logs:
            type: "fluentd"  (9)
            fluentd: {}
      1 The name must be instance.
      2 The cluster logging management state. In some cases, if you change the cluster logging defaults, you must set this to Unmanaged. However, an unmanaged deployment does not receive updates until cluster logging is placed back into a managed state. Placing a deployment back into a managed state might revert any modifications you made.
      3 Settings for configuring Elasticsearch. Using the Custom Resource (CR), you can configure shard replication policy and persistent storage.
      4 Specify the length of time that Elasticsearch should retain each log source. Enter an integer and a time designation: weeks(w), hours(h/H), minutes(m) and seconds(s). For example, 7d for seven days. Logs older than the maxAge are deleted. You must specify a retention policy for each log source or the Elasticsearch indices will not be created for that source.
      5 Specify the number of Elasticsearch nodes. See the note that follows this list.
      6 Enter the name of an existing StorageClass for Elasticsearch storage. For best performance, specify a StorageClass that allocates block storage. If you do not specify a StorageClass, OpenShift Container Platform deploys cluster logging with ephemeral storage only.
      7 Settings for configuring Kibana. Using the CR, you can scale Kibana for redundancy and configure the CPU and memory for your Kibana pods. For more information, see Configuring Kibana.
      8 Settings for configuring the Curator schedule. Curator is used to remove data that is in the Elasticsearch index format prior to OpenShift Container Platform 4.5 and will be removed in a later release.
      9 Settings for configuring Fluentd. Using the CR, you can configure Fluentd CPU and memory limits. For more information, see Configuring Fluentd.

      The maximum number of Elasticsearch master nodes is three. If you specify a nodeCount greater than 3, OpenShift Container Platform creates three Elasticsearch nodes that are Master-eligible nodes, with the master, client, and data roles. The additional Elasticsearch nodes are created as Data-only nodes, using client and data roles. Master nodes perform cluster-wide actions such as creating or deleting an index, shard allocation, and tracking nodes. Data nodes hold the shards and perform data-related operations such as CRUD, search, and aggregations. Data-related operations are I/O-, memory-, and CPU-intensive. It is important to monitor these resources and to add more Data nodes if the current nodes are overloaded.

      For example, if nodeCount=4, the following nodes are created:

      $ oc get deployment
      Example output
      cluster-logging-operator       1/1     1            1           18h
      elasticsearch-cd-x6kdekli-1    1/1     1            0           6m54s
      elasticsearch-cdm-x6kdekli-1   1/1     1            1           18h
      elasticsearch-cdm-x6kdekli-2   1/1     1            0           6m49s
      elasticsearch-cdm-x6kdekli-3   1/1     1            0           6m44s

      The number of primary shards for the index templates is equal to the number of Elasticsearch data nodes.

    2. Create the instance:

      $ oc create -f <file-name>.yaml

      For example:

      $ oc create -f clo-instance.yaml

      This creates the Cluster Logging components, the Elasticsearch Custom Resource and components, and the Kibana interface.

  6. Verify the install by listing the Pods in the openshift-logging project.

    You should see several Pods for Cluster Logging, Elasticsearch, Fluentd, and Kibana similar to the following list:

    oc get pods -n openshift-logging
    Example output
    NAME                                            READY   STATUS    RESTARTS   AGE
    cluster-logging-operator-66f77ffccb-ppzbg       1/1     Running   0          7m
    elasticsearch-cdm-ftuhduuw-1-ffc4b9566-q6bhp    2/2     Running   0          2m40s
    elasticsearch-cdm-ftuhduuw-2-7b4994dbfc-rd2gc   2/2     Running   0          2m36s
    elasticsearch-cdm-ftuhduuw-3-84b5ff7ff8-gqnm2   2/2     Running   0          2m4s
    fluentd-587vb                                   1/1     Running   0          2m26s
    fluentd-7mpb9                                   1/1     Running   0          2m30s
    fluentd-flm6j                                   1/1     Running   0          2m33s
    fluentd-gn4rn                                   1/1     Running   0          2m26s
    fluentd-nlgb6                                   1/1     Running   0          2m30s
    fluentd-snpkt                                   1/1     Running   0          2m28s
    kibana-d6d5668c5-rppqm                          2/2     Running   0          2m39s

Post-installation tasks

If you plan to use Kibana, you must manually create your Kibana index patterns and visualizations to explore and visualize data in Kibana.

Defining Kibana index patterns

An index pattern defines the Elasticsearch indices that you want to visualize. To explore and visualize data in Kibana, you must create an index pattern.

Prerequisites
  • A user must have the cluster-admin role, the cluster-reader role, or both roles to list the infra and audit indices in Kibana.

    For example:

    $ oc auth can-i get pods/logs -n default
    Example output
    yes

    The audit logs are not stored in the internal OpenShift Container Platform Elasticsearch instance by default. To view the audit logs in Kibana, you must use the Log Forwarding API to configure a pipeline that uses the default output for audit logs.

  • Elasticsearch documents must be indexed before you can create index patterns. This is done automatically, but it might take a few minutes in a new or updated cluster.

Procedure

To define index patterns and create visualizations:

  1. Launch Kibana by clicking the Application Launcher app launcher and select Logging.

  2. Create your Kibana index patterns:

    • Regular users must manually create index patterns to see logs for their projects. Users should create a new index pattern named app and use the @timestamp time field to view their container logs.

    • Admin users need to create index patterns for the app, infra, and audit indices using the @timestamp time field.

  3. Create Kibana Visualizations from the new index patterns.

Additional resources