Chapter 14. Configure log-based alerts

Procedure

1. Create an AlertingRule custom resource (CR):

   The following example shows an infrastructure AlertingRule CR:

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: loki-operator-alerts
       namespace: openshift-operators-redhat
       labels:
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "infrastructure"
       groups:
         - name: LokiOperatorHighReconciliationError
           rules:
             - alert: HighPercentageError
               expr: |
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"} |= "error" [1m])) by (job)
                   /
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"}[1m])) by (job)
                   > 0.01
               for: 10s
               labels:
                 severity: critical
               annotations:
                 summary: High Loki Operator Reconciliation Errors
                 description: High Loki Operator Reconciliation Errors

   metadata.namespace

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   metadata.labels

   The labels block must match the LokiStack spec.rules.selector definition.

   spec.tenantID

   AlertingRule CRs for infrastructure tenants are only supported in the openshift-*, kube-\*, or default namespaces.

   expr

   The value for kubernetes_namespace_name: must match the value for metadata.namespace.

   labels.severity

   The value of this mandatory field must be critical, warning, or info.

   annotations.summary

   This field is mandatory.

   annotations.description

   This field is mandatory.

   Example application AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: app-user-workload
       namespace: app-ns
       labels:
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "application"
       groups:
         - name: AppUserWorkloadHighError
           rules:
             - alert:
               expr: |
                 sum(rate({kubernetes_namespace_name="app-ns", kubernetes_pod_name=~"podName.*"} |= "error" [1m])) by (job)
               for: 10s
               labels:
                 severity: critical
               annotations:
                 summary:
                 description:

   metadata.namespace

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   metadata.labels

   The labels block must match the LokiStack spec.rules.selector definition.

   expr

   Value for kubernetes_namespace_name: must match the value for metadata.namespace.

   labels.severity

   The value of this mandatory field must be critical, warning, or info.

   annotations.summary

   The value of this mandatory field is a summary of the rule.

   annotations.description

   The value of this mandatory field is a detailed description of the rule.

2. Apply the AlertingRule CR:

   $ oc apply -f <filename>.yaml