Alert flow architecture

Copy link

Red Hat Edge Manager components emit Prometheus-compatible metrics that describe device health, fleet rollout status, and application deployment state. A Prometheus instance evaluates alert rules against those metrics and sends firing alerts to the bundled Alertmanager. Alertmanager then groups, deduplicates, and routes alerts to one or more receivers that you configure.

When you add a webhook_configs receiver that points to the EDA Controller, the alert flow follows this path:

1. Red Hat Edge Manager components expose metrics on their Prometheus endpoints.
2. Prometheus evaluates alert rules and forwards firing alerts to Alertmanager.
3. Alertmanager applies grouping and inhibition rules to reduce alert noise.
4. Alertmanager routes the grouped alerts and sends an HTTP POST payload to the EDA Controller webhook URL.
5. The ansible.eda.alertmanager source plugin on the EDA Controller receives the payload and converts it into EDA events.
6. An EDA rulebook evaluates the event conditions against its rule set.
7. If a rule matches, the rulebook triggers the associated Ansible Playbook to perform automated remediation.

Each component in this flow is independent. You can modify alert rules, Alertmanager routing, or EDA rulebooks without changing the other components.

The alert payload that Alertmanager sends contains the alert name, severity labels, annotations with a human-readable summary, and the current alert status. Your EDA rulebook conditions can match on any of these fields to decide which playbook to run.

Red Hat Edge Manager Alertmanager infrastructure

Copy link

Red Hat Edge Manager includes a bundled Alertmanager instance as part of the monitoring stack. No additional services are required to enable this integration beyond what Red Hat Edge Manager already provides, though Alertmanager is disabled by default and must be enabled in the MonitoringStack custom resource. Two services work together to provide alert management and external access:

flightctl-alertmanager

The core Alertmanager instance that receives alerts from Prometheus, groups related alerts, applies inhibition and silencing rules, and dispatches notifications to configured receivers. This service manages the full lifecycle of an alert from the moment Prometheus fires it until a receiver acknowledges it.

flightctl-alertmanager-proxy

An authentication proxy that exposes the Alertmanager API through the Red Hat Edge Manager gateway. The proxy handles TLS termination and authentication so that external systems such as the EDA Controller can access alert data securely. You can access alerts through three network paths:

• Port 443 on the Red Hat Edge Manager gateway at the /_/alerts/ path. This path requires gateway authentication and is suitable for browser-based access or API clients that already authenticate with the Red Hat Edge Manager gateway.
• Port 8443 on the integration proxy. This path provides direct TLS access for external systems that need to query the Alertmanager API, such as dashboards or CLI tools.
• Port 9093, the conventional Alertmanager port. This path is available for tools that expect the standard Alertmanager address.

Because both services are part of the Red Hat Edge Manager deployment, you do not need to install or configure a separate Alertmanager instance.

The ansible.eda.alertmanager source plugin

Copy link

The ansible.eda.alertmanager source plugin is a callback-pattern (webhook) event source that is part of the ansible.eda collection in Red Hat Ansible Automation Platform. Unlike polling-based event sources that periodically query an external system, the callback pattern starts an HTTP listener and waits for incoming POST requests. Alertmanager pushes alerts to this listener as soon as they fire.

When the plugin receives an alert payload, it parses the JSON body and converts each alert into an EDA event. The EDA Controller then evaluates those events against the conditions defined in your rulebooks. If a condition matches, the controller launches the specified Ansible Playbook.

The plugin supports the standard Alertmanager webhook payload format. You do not need to transform or adapt the alert data before it reaches EDA. The alert labels, annotations, and status fields from Red Hat Edge Manager are preserved as event fields that you can reference in rulebook conditions.

To use this plugin, you must have Ansible Automation Platform deployed with the EDA Controller component enabled. The ansible.eda collection is included with Red Hat Ansible Automation Platform and does not require a separate installation.

When to use automated alerting

Copy link

Manual monitoring through Grafana dashboards is effective for small deployments where an operator can review alerts and respond in a reasonable time. As fleet sizes grow into hundreds or thousands of devices, the volume of alerts can exceed what a single operator or team can handle. Response times increase, and routine issues that could be resolved automatically consume valuable operator attention.

Consider using this integration when:

• You manage a large number of devices and cannot respond to each alert manually.
• You want to automate remediation for common device issues such as disk space exhaustion, memory pressure, or failed application deployments.
• You need to respond to fleet rollout stalls or update failures without waiting for an operator to notice the problem.
• You want to combine alerting with existing Ansible automation workflows in your organization.
• You need an audit trail of automated responses to device events.
• You want to enforce consistent remediation procedures across your fleet regardless of which operator is on call.

You can use EDA-based automation alongside Grafana dashboards. Automated playbooks handle routine remediation while operators monitor dashboards for issues that require human judgment. This layered approach ensures that predictable problems are resolved immediately while unusual conditions still receive human attention.

Enabling Alertmanager in the monitoring stack

Copy link

To set up the full integration, you must complete the following high-level steps:

1. Deploy the monitoring stack with Alertmanager enabled in the MonitoringStack custom resource.
2. Define Prometheus alert rules that match the device conditions you want to act on.
3. Configure at least one webhook_configs receiver in the Alertmanager configuration that points to your EDA Controller endpoint.
4. Create an EDA rulebook with conditions that match your alert labels and actions that run the appropriate remediation playbooks.

For step-by-step instructions, see Additional resources.