Troubleshoot the observability stack

Resolve common problems when observability services fail to start or Prometheus does not scrape Red Hat Edge Manager metrics after installation.

Services do not start

Check service status and logs:

sudo systemctl status flightctl-prometheus.service
sudo journalctl -u flightctl-prometheus.service -n 50

Common issues:

  • SELinux denials: Run ausearch -m avc -ts recent to inspect denials.
  • Port conflicts: Ensure that ports 3000, 9090, and 8080 are available.
  • Storage permissions: Verify that /var/lib/prometheus and /var/lib/grafana have correct ownership.

Prometheus does not scrape metrics

Inspect the configuration and targets:

cat /etc/flightctl/flightctl-prometheus/prometheus.yml

Check Prometheus targets and verify that Red Hat Edge Manager services are running:

sudo podman exec flightctl-prometheus wget -qO- http://localhost:9090/api/v1/targets 2>/dev/null | jq .
sudo systemctl status flightctl.target