Chapter 4. Install the Red Hat OpenShift Logging Operator

Procedure

1. Create an OperatorGroup object:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: cluster-logging
     namespace: openshift-logging
   spec:
     upgradeStrategy: Default

2. Apply the OperatorGroup object by running the following command:

   $ oc apply -f <filename>.yaml

3. Create a Subscription object for the Red Hat OpenShift Logging Operator:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: cluster-logging
     namespace: openshift-logging
   spec:
     channel: stable-6.6
     installPlanApproval: Automatic
     name: cluster-logging
     source: redhat-operators
     sourceNamespace: openshift-marketplace

   spec.channel

   Specify stable-6.6 as the channel. This must match the channel used for the Loki Operator.

   spec.source

   Specify redhat-operators as the value. If your OpenShift Container Platform cluster is installed on a restricted network, specify the name of the CatalogSource object you created when you configured OLM.

4. Apply the Subscription object by running the following command:

   $ oc apply -f <filename>.yaml

5. Verify that the operator installed successfully by running the following command:

   $ oc get csv -n openshift-logging

   You get an output similar to the following example:

   NAME                                            DISPLAY                      VERSION                       PHASE
   cluster-logging.v6.6.0   Red Hat OpenShift Logging   6.6.0     Succeeded

Procedure

1. Create a service account for the log collector:

   $ oc create sa logging-collector -n openshift-logging

2. Assign the required cluster roles to the service account:

   $ oc adm policy add-cluster-role-to-user logging-collector-logs-writer \
       -z logging-collector -n openshift-logging

   $ oc adm policy add-cluster-role-to-user collect-application-logs \
       -z logging-collector -n openshift-logging

   $ oc adm policy add-cluster-role-to-user collect-infrastructure-logs \
       -z logging-collector -n openshift-logging

   Note

   These three cluster roles are created by the Red Hat OpenShift Logging Operator:

   • logging-collector-logs-writer: Allows writing logs to the LokiStack.
   • collect-application-logs: Allows reading application container logs.
   • collect-infrastructure-logs: Allows reading infrastructure container logs.

   If you also want to collect audit logs, add the collect-audit-logs cluster role.

   Warning

   You must grant the ClusterRoleBindings to the service account before adding the corresponding input type to the ClusterLogForwarder custom resource (CR). If you add an input type to the CR without the required RBAC binding, the entire log collector DaemonSet is deleted and all log collection fails.

Procedure

1. Create a ClusterLogForwarder CR:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: logging-collector
     outputs:
     - name: lokistack-out
       type: lokiStack
       lokiStack:
         target:
           name: logging-loki
           namespace: openshift-logging
         authentication:
           token:
             from: serviceAccount
       tls:
         ca:
           key: service-ca.crt
           configMapName: logging-loki-gateway-ca-bundle
     pipelines:
     - name: infra-app-logs
       inputRefs:
       - application
       - infrastructure
       outputRefs:
       - lokistack-out

   spec.serviceAccount.name

   Specify the service account created in the previous step.

   outputs[0].type

   Select the lokiStack output type to send logs to the LokiStack instance.

   outputs[0].lokiStack.target.name

   Must match the name of the LokiStack instance you created.

   outputs[0].tls.ca

   Required. The LokiStack gateway uses the OpenShift service-serving CA for TLS. Without this block, the collector fails with certificate verify failed: self-signed certificate in certificate chain.

   pipelines[0].inputRefs

   Select the log types to forward. Options: application, infrastructure, and audit.

   Important

   The tls.ca block is required when forwarding logs to a LokiStack in the same cluster. The LokiStack gateway endpoint uses a TLS certificate signed by the cluster’s service-serving CA, which is not in the system truststore. If you omit this block, the ClusterLogForwarder status might show Ready: True, but the collector pods fail to connect at runtime.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. In the OpenShift Container Platform web console Administrator perspective, go to Operators OperatorHub.
2. Type Red Hat OpenShift Logging in the Filter by keyword field. Click Red Hat OpenShift Logging Operator in the list of available Operators, and then click Install.

3. Select stable-6.6 as the Update channel.

   The Red Hat OpenShift Logging Operator must be deployed to the openshift-logging namespace, so the Installation mode and Installed Namespace are already selected. If this namespace does not already exist, it is created for you.

4. Select Enable Operator-recommended cluster monitoring on this namespace.

   This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-logging namespace.

5. For Update approval select Automatic, then click Install.

   Note

   An Operator might display a Failed status before the installation completes. If the Operator installation completes with an InstallSucceeded message, refresh the page.

6. While the Operator installs, create the service account for the log collector:

   1. Click the + button in the top right of the screen to access the Import YAML page.

   2. Enter the following YAML to create the service account:

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: logging-collector
        namespace: openshift-logging

   3. Click Create.

7. Create the required cluster role bindings for the log collector. Click the + button again and enter the following YAML:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: logging-collector:write-logs
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: logging-collector-logs-writer
   subjects:
   - kind: ServiceAccount
     name: logging-collector
     namespace: openshift-logging
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: logging-collector:collect-application
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: collect-application-logs
   subjects:
   - kind: ServiceAccount
     name: logging-collector
     namespace: openshift-logging
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: logging-collector:collect-infrastructure
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: collect-infrastructure-logs
   subjects:
   - kind: ServiceAccount
     name: logging-collector
     namespace: openshift-logging

   Note

   If you also want to collect audit logs, add an additional ClusterRoleBinding with the cluster role collect-audit-logs.

   1. Click Create.
8. Navigate to Operators Installed Operators. Select the Red Hat OpenShift Logging Operator, and under Provided APIs, find the ClusterLogForwarder resource and click Create Instance.

9. Select YAML view, and then use the following template to create a ClusterLogForwarder CR:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: logging-collector
     outputs:
     - name: lokistack-out
       type: lokiStack
       lokiStack:
         target:
           name: logging-loki
           namespace: openshift-logging
         authentication:
           token:
             from: serviceAccount
       tls:
         ca:
           key: service-ca.crt
           configMapName: openshift-service-ca.crt
     pipelines:
     - name: infra-app-logs
       inputRefs:
       - application
       - infrastructure
       outputRefs:
       - lokistack-out

   Important

   The tls.ca block is required. Without it, the collector pods fail to connect to the LokiStack gateway with a certificate verify failed error. The ClusterLogForwarder status might show Ready: True even when this block is missing.

10. Click Create.

Verification

1. In the ClusterLogForwarder tab, verify that you see your ClusterLogForwarder instance.

2. In the Status column, verify that you see the following conditions:

   • observability.openshift.io/Authorized
   • observability.openshift.io/Valid, Ready

Procedure

1. Create a LogFileMetricExporter CR as a YAML file:

   apiVersion: logging.openshift.io/v1alpha1
   kind: LogFileMetricExporter
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     nodeSelector: {}
     resources:
       limits:
         cpu: 500m
         memory: 256Mi
       requests:
         cpu: 200m
         memory: 128Mi
     tolerations: []

   spec.nodeSelector

   Optional: Define the nodes that the exporter pods run on.

   spec.resources

   Optional: Define CPU and memory resource limits and requests for the exporter.

   spec.tolerations

   Optional: Define tolerations that the exporter pods accept.

   Note

   The LogFileMetricExporter is a singleton. The Operator only honors openshift-logging/instance for the namespace and name.

2. Run the following command to apply the LogFileMetricExporter CR:

   $ oc apply -f <filename>.yaml

Procedure

1. Check that collector pods are running on each node:

   $ oc get pods -n openshift-logging -l app.kubernetes.io/component=collector

   You should see one collector pod per node in your cluster.

2. Check the collector logs for errors:

   $ oc logs -n openshift-logging -l app.kubernetes.io/component=collector --tail=20

   Healthy output has only WARN messages about small files. If you see certificate verify failed, verify the tls.ca block in your ClusterLogForwarder.

3. Query logs from the web console:

   Click to Observe Logs and run the following query:

   {log_type="infrastructure"}

   You should see infrastructure log entries from your cluster.