Alertmanager webhook configuration reference

You can configure Alertmanager webhook receivers to forward Red Hat Edge Manager alerts to Red Hat Ansible Automation Platform Event-Driven Ansible (EDA) or other webhook-based endpoints. The configuration is stored in an Alertmanager configuration Secret in your deployment namespace.

Webhook receiver fields
Copy link

The following table describes the fields available in the Alertmanager webhook_configs section.

Table 1. webhook_configs fields
Field Type Required Description

url

String

Yes

The HTTP or HTTPS endpoint URL of the webhook receiver. For EDA Controller, use the format http://<eda_controller_host>:<port>/endpoint, where <port> corresponds to the port configured in the ansible.eda.alertmanager source plugin (default 5000). Use https:// with a tls_config section when the EDA Controller is not on a trusted internal network.

send_resolved

Boolean

No

Whether to send notifications when alerts are resolved. Default is true.

max_alerts

Integer

No

Maximum number of alerts per webhook message. Default is 0 (no limit).

HTTP authentication options
Copy link

You can configure authentication for the webhook endpoint by using the http_config sub-section within webhook_configs.

Table 2. http_config fields
Field Description

basic_auth.username / basic_auth.password

Basic HTTP authentication credentials for the webhook endpoint.

bearer_token / bearer_token_file

Bearer token authentication. Use bearer_token for an inline value or bearer_token_file for a path to a file that contains the token.

tls_config.ca_file

Path to the CA certificate file for verifying the webhook endpoint TLS certificate.

tls_config.cert_file / tls_config.key_file

Client certificate and key for mutual TLS (mTLS) authentication.

tls_config.insecure_skip_verify

Skip TLS verification of the webhook endpoint certificate. Do not use this option in production environments.

Red Hat Edge Manager alert labels
Copy link

Red Hat Edge Manager Prometheus rules attach the following labels to alerts. You can use these labels to build Alertmanager routing rules that match specific alert conditions, or to filter alerts in your EDA rulebooks.

Table 3. Alert labels
Label Description

alertname

The name of the alert rule that fired.

severity

Alert severity level. Valid values are critical, warning, and info.

namespace

The namespace where Red Hat Edge Manager is deployed.

service

The Red Hat Edge Manager service component that generated the metric.

device_id

The identifier of the affected device. Present only for device-specific alerts.

fleet

The name of the fleet. Present only for fleet-specific alerts.

Alert event fields
Copy link

The following table describes the top-level fields available in each event.alert object that the ansible.eda.alertmanager plugin produces.

Table 4. Alert event fields
Field Description

event.alert.status

Current alert state. Value is firing when the alert is active and resolved when it has cleared.

event.alert.startsAt

RFC 3339 timestamp indicating when the alert started firing.

event.alert.annotations.description

Human-readable description of the alert condition, as defined in the Prometheus alert rule.

event.alert.labels.*

All Prometheus labels attached to the alert. See the Red Hat Edge Manager alert labels table above.

Network requirements for Alertmanager access
Copy link

The following table lists the ports and paths through which you can reach the Alertmanager API. The access method you use depends on your deployment topology and network policies.

Table 5. Alertmanager access methods
Access method Port Path Description

Gateway proxy

443

/_/alerts/

Routed through the Red Hat Edge Manager standalone gateway to flightctl-alertmanager-proxy.

Integration proxy

8443

/

Direct access through flightctl-alertmanager-proxy.

Conventional

9093

/

Direct Alertmanager port. Use this method when external monitoring systems connect to Alertmanager without a proxy.

For a complete list of Red Hat Edge Manager network ports, see the Additional resources section.

Example Alertmanager configuration Secret
Copy link

The following example shows a minimal Alertmanager configuration stored in a Kubernetes Secret. This configuration routes critical and warning alerts to a webhook receiver while retaining a default receiver for all other alerts.

Replace <project_name> with the namespace where Red Hat Edge Manager is deployed, and replace <eda_controller_host> with the hostname of your EDA Controller instance:

apiVersion: v1
kind: Secret
metadata:
  name: flightctl-alertmanager-config
  namespace: <project_name>
stringData:
  alertmanager.yaml: |
    route:
      receiver: default
      routes:
        - receiver: eda-webhook
          matchers:
            - severity=~"critical|warning"
          continue: true
    receivers:
      - name: default
      - name: eda-webhook
        webhook_configs:
          - url: "http://<eda_controller_host>:5000/endpoint"
            send_resolved: true

Where:

namespace
The Red Hat OpenShift Container Platform project where Red Hat Edge Manager is deployed.
receiver: default
The fallback receiver for alerts that do not match any route.
matchers
A list of label matchers that select which alerts are forwarded to the webhook. This example matches alerts with a severity label of critical or warning.
continue: true
Continues evaluating subsequent routes after a match, so that the default receiver also receives a copy of matching alerts.
url
The webhook endpoint URL. Replace <eda_controller_host> with your EDA Controller hostname and adjust the port and path as needed.
send_resolved
When set to true, the receiver also sends a notification when a previously firing alert is resolved.