Install the observability stack in disconnected environments

Mirror the Prometheus and Grafana container images and the flightctl-observability RPM package, then install and start the observability stack on an air-gapped Red Hat Edge Manager management host.

Before you begin

About this task

The standard offline installation bundle created by mirror-images excludes Prometheus and Grafana container images because flightctl-observability is optional. If you plan to run the observability stack on an air-gapped Red Hat Edge Manager management host, mirror those images manually and include the flightctl-observability RPM when you build the bundle.

Important

Grafana data sources, dashboards, and Prometheus alerting rules ship with the flightctl-observability RPM as defaults for Red Hat Edge Manager metrics. Organization-specific customization (additional dashboards, external data sources, and alerting integrations) must be configured manually. There is no automated mechanism to configure these in disconnected environments at this time.

Procedure

  1. On the prep machine, identify the Prometheus and Grafana images that match the --variant value you use for the main bundle (rhem-el9 or rhem-el10):
    Table 1. Observability container images
    Image Variant

    registry.redhat.io/rhel9/grafana:9.7-1766395319

    rhem-el9, rhem-el10

    registry.redhat.io/rhacm2/prometheus-rhel9:v2.14.2-1

    rhem-el9, rhem-el10
  2. When you run mirror-images on the prep machine, add flightctl-observability to --rpm-packages so the RPM and its dependencies are bundled with the core service packages. See mirror-images flag reference and Pinning bundle contents to a release version for version pinning: 
    mirror-images \
    --variant <variant> \
    --bundle ~/rhem-bundle.tar.gz \
    --bundle-rpms \
    --rpm-createrepo \
    --rpm-packages flightctl-services,flightctl-observability

    Replace <variant> with rhem-el9 or rhem-el10. When you pin a release, include version suffixes on both package names (for example flightctl-services-1.3.0,flightctl-observability-1.3.0).

  3. On the connected prep machine, copy the observability container images to a local directory by using skopeo. The following example uses the rhem-el9 Red Hat registry images: 
    skopeo copy docker://registry.redhat.io/rhel9/grafana:9.7-1766395319 \
    dir:~/obs-images/rhel9-grafana:9.7-1766395319
skopeo copy docker://registry.redhat.io/rhacm2/prometheus-rhel9:v2.14.2-1 \
    dir:~/obs-images/rhacm2-prometheus-rhel9:v2.14.2-1

    Adjust image references to match your variant and the tags from the previous step.

  4. Archive the directory and transfer it to the air-gapped target alongside the main bundle: 
    tar -czf ~/observability-images.tar.gz -C ~/obs-images .
scp ~/observability-images.tar.gz <user>@<target_host>:~/
  5. On the target machine, extract the archive and push each image into the local registry that Red Hat Edge Manager services use (typically localhost:5000 after Installing from an offline bundle): 
    mkdir ~/obs-images
tar -xzf ~/observability-images.tar.gz -C ~/obs-images

skopeo copy --dest-tls-verify=false \
    "dir:$HOME/obs-images/rhel9-grafana:9.7-1766395319" \
    "docker://localhost:5000/rhel9/grafana:9.7-1766395319"
skopeo copy --dest-tls-verify=false \
    "dir:$HOME/obs-images/rhacm2-prometheus-rhel9:v2.14.2-1" \
    "docker://localhost:5000/rhacm2/prometheus-rhel9:v2.14.2-1"

    Replace directory names and destination paths with the values that match your mirrored images. Ensure /etc/containers/registries.conf redirects registry.redhat.io to your local registry. If you followed Installing from an offline bundle, this configuration is already in place.

  6. Install flightctl-observability from the bundle RPM tree, then start the observability stack: 
    cd ~/rhem-bundle
sudo dnf install -y --nogpgcheck --disablerepo='*' rpms/flightctl-observability*.rpm rpms/*.rpm

sudo systemctl enable --now flightctl-observability.target