×

You can view the status of the Red Hat OpenShift Logging Operator and other logging components.

Viewing the status of the Red Hat OpenShift Logging Operator

You can view the status of the Red Hat OpenShift Logging Operator.

Prerequisites
  • The Red Hat OpenShift Logging Operator and OpenShift Elasticsearch Operator are installed.

Procedure
  1. Change to the openshift-logging project by running the following command:

    $ oc project openshift-logging
  2. Get the ClusterLogging instance status by running the following command:

    $ oc get clusterlogging instance -o yaml
    Example output
    apiVersion: logging.openshift.io/v1
    kind: ClusterLogging
    # ...
    status:  (1)
      collection:
        logs:
          fluentdStatus:
            daemonSet: fluentd  (2)
            nodes:
              collector-2rhqp: ip-10-0-169-13.ec2.internal
              collector-6fgjh: ip-10-0-165-244.ec2.internal
              collector-6l2ff: ip-10-0-128-218.ec2.internal
              collector-54nx5: ip-10-0-139-30.ec2.internal
              collector-flpnn: ip-10-0-147-228.ec2.internal
              collector-n2frh: ip-10-0-157-45.ec2.internal
            pods:
              failed: []
              notReady: []
              ready:
              - collector-2rhqp
              - collector-54nx5
              - collector-6fgjh
              - collector-6l2ff
              - collector-flpnn
              - collector-n2frh
      logstore: (3)
        elasticsearchStatus:
        - ShardAllocationEnabled:  all
          cluster:
            activePrimaryShards:    5
            activeShards:           5
            initializingShards:     0
            numDataNodes:           1
            numNodes:               1
            pendingTasks:           0
            relocatingShards:       0
            status:                 green
            unassignedShards:       0
          clusterName:             elasticsearch
          nodeConditions:
            elasticsearch-cdm-mkkdys93-1:
          nodeCount:  1
          pods:
            client:
              failed:
              notReady:
              ready:
              - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
            data:
              failed:
              notReady:
              ready:
              - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
            master:
              failed:
              notReady:
              ready:
              - elasticsearch-cdm-mkkdys93-1-7f7c6-mjm7c
    visualization:  (4)
        kibanaStatus:
        - deployment: kibana
          pods:
            failed: []
            notReady: []
            ready:
            - kibana-7fb4fd4cc9-f2nls
          replicaSets:
          - kibana-7fb4fd4cc9
          replicas: 1
    1 In the output, the cluster status fields appear in the status stanza.
    2 Information on the Fluentd pods.
    3 Information on the Elasticsearch pods, including Elasticsearch cluster health, green, yellow, or red.
    4 Information on the Kibana pods.

Example condition messages

The following are examples of some condition messages from the Status.Nodes section of the ClusterLogging instance.

A status message similar to the following indicates a node has exceeded the configured low watermark and no shard will be allocated to this node:

Example output
  nodes:
  - conditions:
    - lastTransitionTime: 2019-03-15T15:57:22Z
      message: Disk storage usage for node is 27.5gb (36.74%). Shards will be not
        be allocated on this node.
      reason: Disk Watermark Low
      status: "True"
      type: NodeStorage
    deploymentName: example-elasticsearch-clientdatamaster-0-1
    upgradeStatus: {}

A status message similar to the following indicates a node has exceeded the configured high watermark and shards will be relocated to other nodes:

Example output
  nodes:
  - conditions:
    - lastTransitionTime: 2019-03-15T16:04:45Z
      message: Disk storage usage for node is 27.5gb (36.74%). Shards will be relocated
        from this node.
      reason: Disk Watermark High
      status: "True"
      type: NodeStorage
    deploymentName: cluster-logging-operator
    upgradeStatus: {}

A status message similar to the following indicates the Elasticsearch node selector in the CR does not match any nodes in the cluster:

Example output
    Elasticsearch Status:
      Shard Allocation Enabled:  shard allocation unknown
      Cluster:
        Active Primary Shards:  0
        Active Shards:          0
        Initializing Shards:    0
        Num Data Nodes:         0
        Num Nodes:              0
        Pending Tasks:          0
        Relocating Shards:      0
        Status:                 cluster health unknown
        Unassigned Shards:      0
      Cluster Name:             elasticsearch
      Node Conditions:
        elasticsearch-cdm-mkkdys93-1:
          Last Transition Time:  2019-06-26T03:37:32Z
          Message:               0/5 nodes are available: 5 node(s) didn't match node selector.
          Reason:                Unschedulable
          Status:                True
          Type:                  Unschedulable
        elasticsearch-cdm-mkkdys93-2:
      Node Count:  2
      Pods:
        Client:
          Failed:
          Not Ready:
            elasticsearch-cdm-mkkdys93-1-75dd69dccd-f7f49
            elasticsearch-cdm-mkkdys93-2-67c64f5f4c-n58vl
          Ready:
        Data:
          Failed:
          Not Ready:
            elasticsearch-cdm-mkkdys93-1-75dd69dccd-f7f49
            elasticsearch-cdm-mkkdys93-2-67c64f5f4c-n58vl
          Ready:
        Master:
          Failed:
          Not Ready:
            elasticsearch-cdm-mkkdys93-1-75dd69dccd-f7f49
            elasticsearch-cdm-mkkdys93-2-67c64f5f4c-n58vl
          Ready:

A status message similar to the following indicates that the requested PVC could not bind to PV:

Example output
      Node Conditions:
        elasticsearch-cdm-mkkdys93-1:
          Last Transition Time:  2019-06-26T03:37:32Z
          Message:               pod has unbound immediate PersistentVolumeClaims (repeated 5 times)
          Reason:                Unschedulable
          Status:                True
          Type:                  Unschedulable

A status message similar to the following indicates that the Fluentd pods cannot be scheduled because the node selector did not match any nodes:

Example output
Status:
  Collection:
    Logs:
      Fluentd Status:
        Daemon Set:  fluentd
        Nodes:
        Pods:
          Failed:
          Not Ready:
          Ready:

Viewing the status of logging components

You can view the status for a number of logging components.

Prerequisites
  • The Red Hat OpenShift Logging Operator and OpenShift Elasticsearch Operator are installed.

Procedure
  1. Change to the openshift-logging project.

    $ oc project openshift-logging
  2. View the status of logging environment:

    $ oc describe deployment cluster-logging-operator
    Example output
    Name:                   cluster-logging-operator
    
    ....
    
    Conditions:
      Type           Status  Reason
      ----           ------  ------
      Available      True    MinimumReplicasAvailable
      Progressing    True    NewReplicaSetAvailable
    
    ....
    
    Events:
      Type    Reason             Age   From                   Message
      ----    ------             ----  ----                   -------
      Normal  ScalingReplicaSet  62m   deployment-controller  Scaled up replica set cluster-logging-operator-574b8987df to 1----
  3. View the status of the logging replica set:

    1. Get the name of a replica set:

      Example output
      $ oc get replicaset
      Example output
      NAME                                      DESIRED   CURRENT   READY   AGE
      cluster-logging-operator-574b8987df       1         1         1       159m
      elasticsearch-cdm-uhr537yu-1-6869694fb    1         1         1       157m
      elasticsearch-cdm-uhr537yu-2-857b6d676f   1         1         1       156m
      elasticsearch-cdm-uhr537yu-3-5b6fdd8cfd   1         1         1       155m
      kibana-5bd5544f87                         1         1         1       157m
    2. Get the status of the replica set:

      $ oc describe replicaset cluster-logging-operator-574b8987df
      Example output
      Name:           cluster-logging-operator-574b8987df
      
      ....
      
      Replicas:       1 current / 1 desired
      Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
      
      ....
      
      Events:
        Type    Reason            Age   From                   Message
        ----    ------            ----  ----                   -------
        Normal  SuccessfulCreate  66m   replicaset-controller  Created pod: cluster-logging-operator-574b8987df-qjhqv----