Chapter 6. Set up log forwarding

Procedure

1. Create a service account for the collector. This example creates the service account in the openshift-logging namespace, but you can deploy the collector in any namespace by running the following command:

   $ oc create sa collector -n openshift-logging

2. Grant the service account permission to collect application logs by running the following command:

   $ oc adm policy add-cluster-role-to-user collect-application-logs \
     system:serviceaccount:openshift-logging:collector

3. Grant the service account permission to collect infrastructure logs by running the following command:

   $ oc adm policy add-cluster-role-to-user collect-infrastructure-logs \
     system:serviceaccount:openshift-logging:collector

4. Optional: Grant the service account permission to collect audit logs by running the following command:

   $ oc adm policy add-cluster-role-to-user collect-audit-logs \
     system:serviceaccount:openshift-logging:collector

   Note

   You can add audit log collection later. If you do, grant this ClusterRoleBinding before you add audit to the inputRefs list in your ClusterLogForwarder CR.

   Considerations for audit logs:

   • Volume and Cost: Audit logs are significantly more voluminous than application or infrastructure logs. In cloud environments, this can lead to unexpected storage and data transfer costs.
   • Sensitivity: Audit logs contain a detailed record of API activity. To protect sensitive metadata and manage data volume, consider implementing prune filters to remove specific fields before the logs are forwarded to their destination.

Procedure

1. Define the ClusterLogForwarder CR. Use the following example to forward application and infrastructure logs to an in-cluster LokiStack.

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: collector
     outputs:
     - name: default-lokistack
       type: lokiStack
       lokiStack:
         authentication:
           token:
             from: serviceAccount
         target:
           name: logging-loki
           namespace: openshift-logging
       tls:
         ca:
           configMapName: logging-loki-gateway-ca-bundle
           key: service-ca.crt
     pipelines:
     - name: default-logstore
       inputRefs:
       - application
       - infrastructure
       outputRefs:
       - default-lokistack

   metadata.name

   The name must be instance. It is a singleton.

   spec.serviceAccount.name

   The service account that the collector uses to authenticate. This service account must have ClusterRoleBindings for each log type listed in inputRefs.

   spec.outputs[].type

   Set to lokiStack for the in-cluster LokiStack. This type uses the LokiStack gateway for authentication and multi-tenancy.

   spec.outputs[].lokiStack.target.name

   The name of your LokiStack CR. This value must strictly match the name of the LokiStack you want to forward logs to, and is not bound to a specific default.

   spec.outputs[].tls.ca.configMapName

   The ConfigMap that has the certificate authority (CA) bundle for the LokiStack gateway. Use logging-loki-gateway-ca-bundle.

   spec.outputs[].tls.ca.key

   The key in the ConfigMap that has the CA certificate. For the service-signed certificate, use service-ca.crt.

   Note

   You must include the tls.ca block in the output configuration when forwarding to LokiStack. Without it, the collector cannot verify the LokiStack gateway certificate. The ClusterLogForwarder reports Ready, but the collector fails silently and no logs reach LokiStack.

   spec.pipelines[].inputRefs

   The log types to collect. Add only log types for which the service account has ClusterRoleBindings.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

3. Verify that the ClusterLogForwarder status shows Ready by running the following command:

   $ oc get clusterlogforwarder instance -n openshift-logging \
     -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

   The output should show True. If the output shows False, check the status.conditions for error messages:

   $ oc get clusterlogforwarder instance -n openshift-logging \
     -o jsonpath='{.status.conditions}' | python3 -m json.tool