Managing alerts as an Administrator

From the Administrator perspective of the OpenShift Container Platform web console, go to Observe → Alerting → Alerts.

For the alert that you want to silence, click and select Silence alert to open the Silence alert page with a default configuration for the chosen alert.

Optional: Change the default configuration details for the silence.

To save the silence, click Silence.

From the Administrator perspective of the OpenShift Container Platform web console, go to Observe → Alerting → Silences.

Click Create silence.

On the Create silence page, set the schedule, duration, and label details for an alert.

To create silences for alerts that match the labels that you entered, click Silence.

From the Administrator perspective of the OpenShift Container Platform web console, go to Observe → Alerting → Silences.

For the silence you want to modify, click and select Edit silence.

On the Edit silence page, make changes and click Silence. Doing so expires the existing silence and creates one with the updated configuration.

Go to Observe → Alerting → Silences.

For the silence or silences you want to expire, select the checkbox in the corresponding row.

Click Expire 1 silence to expire a single selected silence or Expire <n> silences to expire multiple selected silences, where <n> is the number of silences you selected.

Create a new YAML configuration file named example-alerting-rule.yaml.

Add an AlertingRule resource to the YAML file. The following example creates a new alerting rule named example, similar to the default Watchdog alert:

apiVersion: monitoring.openshift.io/v1
kind: AlertingRule
metadata:
  name: example
  namespace: openshift-monitoring (1)
spec:
  groups:
  - name: example-rules
    rules:
    - alert: ExampleAlert (2)
      for: 1m (3)
      expr: vector(1) (4)
      labels:
        severity: warning (5)
      annotations:
        message: This is an example alert. (6)

Apply the configuration file to the cluster:

$ oc apply -f example-alerting-rule.yaml

Create a new YAML configuration file named example-modified-alerting-rule.yaml.

Add an AlertRelabelConfig resource to the YAML file. The following example modifies the severity setting to critical for the default platform watchdog alerting rule:

apiVersion: monitoring.openshift.io/v1
kind: AlertRelabelConfig
metadata:
  name: watchdog
  namespace: openshift-monitoring (1)
spec:
  configs:
  - sourceLabels: [alertname,severity] (2)
    regex: "Watchdog;none" (3)
    targetLabel: severity (4)
    replacement: critical (5)
    action: Replace (6)

Apply the configuration file to the cluster:

$ oc apply -f example-modified-alerting-rule.yaml

Create a YAML file for alerting rules. In this example, it is called example-app-alerting-rule.yaml.

Add an alerting rule configuration to the YAML file. The following example creates a new alerting rule named example-alert. The alerting rule fires an alert when the version metric exposed by the sample service becomes 0:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-alert
  namespace: ns1
spec:
  groups:
  - name: example
    rules:
    - alert: VersionAlert (1)
      for: 1m (2)
      expr: version{job="prometheus-example-app"} == 0 (3)
      labels:
        severity: warning (4)
      annotations:
        message: This is an example alert. (5)

Apply the configuration file to the cluster:

$ oc apply -f example-app-alerting-rule.yaml

Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

$ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

Configure projects in which you want to create alerting rules that are not bound to a specific project:

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    namespacesWithoutLabelEnforcement: [ <namespace> ] (1)
    # ...

Create a YAML file for alerting rules. In this example, it is called example-cross-project-alerting-rule.yaml.

Add an alerting rule configuration to the YAML file. The following example creates a new cross-project alerting rule called example-security. The alerting rule fires when a user project does not enforce the restricted pod security policy:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-security
namespace: ns1 (1)
spec:
  groups:
    - name: pod-security-policy
      rules:
        - alert: "ProjectNotEnforcingRestrictedPolicy" (2)
          for: 5m (3)
          expr: kube_namespace_labels{namespace!~"(openshift|kube).*|default",label_pod_security_kubernetes_io_enforce!="restricted"} (4)
          annotations:
            message: "Restricted policy not enforced. Project {{ $labels.namespace }} does not enforce the restricted pod security policy." (5)
          labels:
            severity: warning (6)

Apply the configuration file to the cluster:

$ oc apply -f example-cross-project-alerting-rule.yaml

From the Administrator perspective of the OpenShift Container Platform web console, go to Observe → Alerting → Alerting rules.

Select the Platform and User sources in the Filter drop-down menu.

Edit the cluster-monitoring-config config map in the openshift-monitoring project:

$ oc -n openshift-monitoring edit configmap cluster-monitoring-config

In the cluster-monitoring-config config map, disable the option to create cross-project alerting rules by setting the rulesWithoutLabelEnforcementAllowed value under data/config.yaml/userWorkload to false:

kind: ConfigMap
apiVersion: v1
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    userWorkload:
      rulesWithoutLabelEnforcementAllowed: false
    # ...

Save the file to apply the changes.