Restore by using quadlets and flightctl-restore

Typical RPM installations run Red Hat Edge Manager application components as separate systemd units backed by Podman quadlets. The PostgreSQL and KV store containers keep running until you intentionally restart them.

Before you begin

  • Prerequisites for the restore procedure are satisfied.
  • Root or sudo access to the host where the quadlets run.
  • A tested backup of the flightctl PostgreSQL database.
  • flightctl-restore installed on the host and matching the Red Hat Edge Manager server version.

About this task

Note

For recovery from a flightctl-backup archive, use the full server restore procedure. For more information about restoring the full server on Red Hat Enterprise Linux, see Additional resources. The steps below apply when you restore the database manually and then run flightctl-restore without an archive argument for device preparation.

This sequence matches the quadlet layout: stop application services only, restore the flightctl database, supply credentials, and expose ports locally. Then run the flightctl-restore binary, remove temporary port publishing, and start application services again.

Procedure

  1. Verify that flightctl-restore matches the Red Hat Edge Manager server version:

    Run flightctl version from a host where the CLI can reach the Red Hat Edge Manager API (use flightctl login if needed). The command reports Client Version (the CLI binary) and, when connected, Server Version (the running Red Hat Edge Manager instance). Run flightctl-restore version to report the restore binary version.

    flightctl version
flightctl-restore version

    Compare the output: Client Version, Server Version, and the flightctl-restore version must all report the same Red Hat Edge Manager release. If Server Version is missing or any version differs, update the mismatched binaries before you continue. For more information about prerequisites for backup and restore, see Additional resources.

    Important

    Cross-version restore is not supported. The CLI client, the running server, and flightctl-restore must be on the same release before you continue.
  2. Stop the Red Hat Edge Manager application services so they do not write to the database during restore. Do not stop flightctl-db.service, flightctl-kv.service, or other database or KV units; they must keep running while you restore data and run flightctl-restore. 
    # Stop only application services (keep database and KV store running)
sudo systemctl stop flightctl-gateway.service
sudo systemctl stop flightctl-api.service
sudo systemctl stop flightctl-worker.service
sudo systemctl stop flightctl-periodic.service
sudo systemctl stop flightctl-alert-exporter.service
sudo systemctl stop flightctl-alertmanager-proxy.service
sudo systemctl stop flightctl-telemetry-gateway.service
sudo systemctl stop flightctl-pam-issuer.service
sudo systemctl stop flightctl-cli-artifacts.service
sudo systemctl stop flightctl-alertmanager.service
sudo systemctl stop flightctl-imagebuilder-api.service
sudo systemctl stop flightctl-imagebuilder-worker.service
sudo systemctl stop flightctl-ui.service
    Note

    If systemctl reports Unknown unit for a service, your host might not ship that component (for example, image builder or UI). Skip that line. Do not run systemctl stop flightctl.target here; that would stop the database and KV services as well.
  3. Confirm that the application units you stopped are inactive: 
    sudo systemctl status flightctl-api.service
systemctl is-active flightctl-api.service
  4. Restore the flightctl PostgreSQL database by using the method that matches your backup (for example, pg_restore, psql with a SQL dump, or storage-level recovery). Ensure the database is consistent and reachable from the deployment before you run flightctl-restore.
  5. Retrieve the database application password from the Podman secret: 
    DB_APP_PASSWORD=$(sudo podman secret inspect flightctl-postgresql-user-password --showsecret | jq -r '.[0].SecretData')
echo "Database password retrieved successfully"
  6. Retrieve the KV store password from the Podman secret: 
    KV_PASSWORD=$(sudo podman secret inspect flightctl-kv-password --showsecret | jq -r '.[0].SecretData')
echo "KV store credentials retrieved successfully"
  7. Optional: verify database and KV connectivity from the host through the running containers:

    Database: verify readiness:

    sudo podman exec flightctl-db pg_isready -U postgres

    To connect to the database for additional verification (optional):

    sudo podman exec -it flightctl-db psql -U flightctl_app -d flightctl

    KV store: on Red Hat Enterprise Linux 9 you can use redis-cli; on Red Hat Enterprise Linux 10 use valkey-cli inside the container, for example:

    sudo podman exec flightctl-kv redis-cli ping
  8. Publish the database and KV ports on localhost so flightctl-restore can reach them. The database and KV containers use a private network by default; use temporary systemd drop-in files to add port publishing, then reload and restart only those services: 
    DB_CONTAINER_FILE=$(systemctl show flightctl-db.service -p SourcePath --value)
KV_CONTAINER_FILE=$(systemctl show flightctl-kv.service -p SourcePath --value)
DB_DROPIN_DIR="${DB_CONTAINER_FILE}.d"
KV_DROPIN_DIR="${KV_CONTAINER_FILE}.d"
sudo mkdir -p "$DB_DROPIN_DIR" "$KV_DROPIN_DIR"

sudo tee "$DB_DROPIN_DIR/10-publish-port.conf" > /dev/null <<'EOF'
[Container]
PublishPort=5432:5432
EOF

sudo tee "$KV_DROPIN_DIR/10-publish-port.conf" > /dev/null <<'EOF'
[Container]
PublishPort=6379:6379
EOF

sudo systemctl daemon-reload
sudo systemctl restart flightctl-db.service flightctl-kv.service

    Verify listening ports and basic connectivity (adjust the KV client for your Red Hat Enterprise Linux major version):

    ss -tlnp | grep -E ':5432|:6379' || true
pg_isready -h localhost -p 5432
REDISCLI_AUTH="$KV_PASSWORD" redis-cli -h localhost -p 6379 ping

    On Red Hat Enterprise Linux 10, use VALKEYCLI_AUTH with valkey-cli instead of REDISCLI_AUTH with redis-cli if that matches your environment.

  9. Run flightctl-restore with the database and KV passwords (run from the directory that contains the binary if you do not use a full path): 
    DB_PASSWORD="$DB_APP_PASSWORD" KV_PASSWORD="$KV_PASSWORD" flightctl-restore

    Monitor the command output until it completes successfully. If it fails or logs per-device KV errors, keep application services stopped. For more information about recovering from a failed or partial restore, see Additional resources before you remove port publishing or start application services.

  10. Remove the temporary port publishing drop-ins and restart the database and KV services: 
    DB_CONTAINER_FILE=$(systemctl show flightctl-db.service -p SourcePath --value)
KV_CONTAINER_FILE=$(systemctl show flightctl-kv.service -p SourcePath --value)
DB_DROPIN_DIR="${DB_CONTAINER_FILE}.d"
KV_DROPIN_DIR="${KV_CONTAINER_FILE}.d"
sudo rm -f "$DB_DROPIN_DIR/10-publish-port.conf" "$KV_DROPIN_DIR/10-publish-port.conf"
sudo rmdir "$DB_DROPIN_DIR" 2>/dev/null || true
sudo rmdir "$KV_DROPIN_DIR" 2>/dev/null || true
sudo systemctl daemon-reload
sudo systemctl restart flightctl-db.service flightctl-kv.service
  11. Start the application services again (same set you stopped; omit units your host does not use): 
    sudo systemctl start flightctl-api.service
sudo systemctl start flightctl-worker.service
sudo systemctl start flightctl-periodic.service
sudo systemctl start flightctl-alert-exporter.service
sudo systemctl start flightctl-alertmanager-proxy.service
sudo systemctl start flightctl-telemetry-gateway.service
sudo systemctl start flightctl-pam-issuer.service
sudo systemctl start flightctl-cli-artifacts.service
sudo systemctl start flightctl-alertmanager.service
sudo systemctl start flightctl-imagebuilder-api.service
sudo systemctl start flightctl-imagebuilder-worker.service
sudo systemctl start flightctl-ui.service
sudo systemctl start flightctl-gateway.service

    Verify with sudo systemctl status on each unit and sudo podman ps --filter "name=flightctl-".

  12. Confirm that the API responds and that inventory looks correct in the Red Hat Edge Manager web console or with flightctl CLI commands.

Results

  • flightctl API calls succeed.
  • Devices move from AwaitingReconnect to online where expected.
  • Application services are active and running.