Configure Alertmanager to forward alerts to Event-Driven Ansible

You can integrate Alertmanager with the Red Hat Ansible Automation Platform EDA Controller so that device alerts automatically trigger remediation playbooks instead of requiring manual operator intervention.

Before you begin

  • Red Hat Ansible Automation Platform 2.5 with the EDA Controller component installed and accessible over HTTPS.
  • Red Hat Edge Manager deployed on Red Hat OpenShift Container Platform with the monitoring stack (Cluster Observability Operator and MonitoringStack CR) from Deploying the monitoring stack.
  • The OpenShift CLI (oc) installed and authenticated with cluster-admin permissions.
  • Network connectivity from the Alertmanager pod to the EDA Controller endpoint.

Procedure

  1. Create a Secret that contains the Alertmanager configuration with a webhook receiver pointing to the EDA Controller.

    In the following command, replace <project_name> with the namespace where Red Hat Edge Manager is installed, <eda_controller_host> with the EDA Controller hostname or IP address, and <port> with the port number of the EDA webhook listener (typically 5000):

    $ oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: flightctl-alertmanager-config
  namespace: <project_name>
stringData:
  alertmanager.yaml: |
    route:
      receiver: default
      group_by: ['alertname', 'namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      routes:
        - receiver: eda-webhook
          matchers:
            - severity=~"critical|warning"
          continue: true
    receivers:
      - name: default
      - name: eda-webhook
        webhook_configs:
          - url: "http://<eda_controller_host>:<port>/endpoint"
            send_resolved: true
EOF

    Where:

    route.group_by
    Groups alerts by the alertname and namespace labels before sending them to the receiver.
    route.routes[0].matchers
    Filters alerts so that only those with a severity of critical or warning are forwarded to the EDA Controller. You can adjust the regular expression to include other severity levels.
    route.routes[0].continue
    When set to true, alerts that match this route are also evaluated against subsequent routes. This ensures alerts are sent to both the eda-webhook and default receivers.
    webhook_configs[0].url
    The HTTP endpoint of the EDA Controller where Alertmanager sends alert payloads. Replace <eda_controller_host> and <port> with values that match your EDA Controller deployment.
    webhook_configs[0].send_resolved
    When set to true, Alertmanager sends a notification when an alert is resolved in addition to when it fires.
    Warning

    The example uses http:// for the webhook URL. Use http:// only when Alertmanager and EDA Controller communicate over a trusted network or the same cluster. For production environments, use https:// and configure TLS authentication in the http_config section. See Alertmanager webhook configuration reference for TLS options.
  2. Patch the existing flightctl-monitoring-stackMonitoringStack resource to enable Alertmanager and reference the configuration secret: 
    $ oc patch monitoringstack flightctl-monitoring-stack \
  -n <project_name> \
  --type=merge \
  -p '{
    "spec": {
      "alertmanagerConfig": {
        "disabled": false,
        "alertmanagerConfigSecretName": "flightctl-alertmanager-config"
      }
    }
  }'

    This command changes alertmanagerConfig.disabled from true to false and sets alertmanagerConfigSecretName to the secret you created in the previous step. The Cluster Observability Operator detects the change and deploys an Alertmanager pod in the <project_name> namespace.

    Note

    To disable Alertmanager later, run the same oc patch command with "disabled": true.
  3. Verify that the Alertmanager pod is running: 
    $ oc wait --for=condition=Ready pod -l app.kubernetes.io/name=alertmanager \
  -n <project_name> --timeout=120s

    The command returns when the Alertmanager pod reaches the Ready state. If the pod does not become ready within 120 seconds, check the pod events by running oc describe pod -l app.kubernetes.io/name=alertmanager -n <project_name>.

  4. Create a Git repository or a directory in an existing repository that contains your EDA rulebooks.

    The following example shows a minimal rulebook that listens for Alertmanager webhooks and runs a playbook when an alert fires:

    ---
- name: Listen for RHEM alerts
  hosts: all
  sources:
    - ansible.eda.alertmanager:
        host: 0.0.0.0
        port: 5000
  rules:
    - name: Log all alerts
      condition: event.alert.status == "firing"
      action:
        run_playbook:
          name: playbooks/handle_alert.yml
          extra_vars:
            alert_name: "{{ event.alert.labels.alertname }}"
            device_id: "{{ event.alert.labels.device_id }}"

    The ansible.eda.alertmanager source plugin starts an HTTP listener on the specified host and port. When Alertmanager sends an alert payload, the plugin parses the JSON body and converts each alert into an EDA event. Alertmanager sends a single HTTP POST containing an alerts array that may include multiple alerts grouped together. The plugin splits this array and enqueues one event per alert, so each event your rulebook evaluates contains a single alert object at event.alert, not an array. This means rulebook conditions reference event.alert.labels.alertname, not event.alerts[0].labels.alertname. The condition field evaluates the event data, and the action field specifies the playbook to run when the condition is met. Customize the extra_vars mapping to pass alert labels that are relevant to your remediation playbooks.

  5. Create a rulebook activation in the Red Hat Ansible Automation Platform web interface:
    1. Navigate to Event-Driven Ansible Rulebook Activations.
    2. Click Create rulebook activation.
    3. Set the following fields:
      • Name: A descriptive name, for example RHEM Alert Handler.
      • Project: Select or create a project that points to the Git repository from the previous step.
      • Rulebook: Select the rulebook file.
      • Decision environment: Select a decision environment that includes the ansible.eda collection.
      • Restart policy: Set to Always for production use.
    4. Click Create rulebook activation.
    5. Verify that the activation status shows Running.

Results

Verify the end-to-end alert flow by creating a test alert and confirming that the EDA Controller receives it.

  1. Create a test alert by port-forwarding to the Alertmanager pod and sending a request to the Alertmanager API from your local workstation: 
    $ oc port-forward -n <project_name> \
  svc/flightctl-alertmanager 9093:9093 &
$ curl -X POST http://localhost:9093/api/v2/alerts \
  -H 'Content-Type: application/json' \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "device_id": "test-device-001"
    },
    "annotations": {
      "description": "Test alert for EDA integration verification"
    }
  }]'

    After you confirm the alert, stop the port-forward process by running kill %1 or pressing Ctrl+C in the terminal where the port-forward is running.

  2. In the Red Hat Ansible Automation Platform web interface, navigate to Event-Driven Ansible Rulebook Activations and select your activation.
  3. Open the History tab and verify that a new event is displayed with the test alert details.
  4. The test alert auto-resolves within approximately 5 minutes after the last firing notification. No manual cleanup is required.