Troubleshoot backup and restore

Resolve common flightctl-backup and flightctl-restore failures by matching the symptom to the recommended action. Exact messages can vary by release; compare CLI output to flightctl-backup --help and flightctl-restore --help for your version.

Symptom Likely cause What to do

sha256sum -c fails

Corrupt or partial archive copy

Re-copy the archive from the source; re-run backup if the source file is damaged

Restore reports checksum mismatch before extraction

Archive or .sha256 file altered

Do not restore; obtain a known-good archive from backup storage

Restore reports deployment type mismatch

Archive from Podman host restored on Red Hat OpenShift Container Platform, or the reverse

Take a new backup on the target deployment type, or rebuild the environment to match the archive metadata

Backup fails: pg_dump not found

Client tools not installed on the backup execution host

Install PostgreSQL client packages so pg_dump is in PATH

Backup or restore fails: database connection error

Wrong credentials, database not running, or network policy blocking access

Verify database pod or container is running; confirm passwords from Secrets or Podman secrets; on Red Hat OpenShift Container Platform, check port-forward and namespace

Backup completes but db/dump.sql is absent

External PostgreSQL configured

Follow printed instructions and your DBA runbook. For more information about external PostgreSQL databases, see Additional resources.

Restore fails: version mismatch

flightctl, flightctl-backup, or flightctl-restore does not match server version

Install or replace mismatched binaries from the same Red Hat Edge Manager release as the server. For more information about prerequisites for backup and restore, see Additional resources.

Devices do not reconnect after restore

PKI not restored, wrong archive, or network/DNS change

Verify pki/ was restored; confirm server URL and certificates unchanged. For more information about post-restore device status changes, see Additional resources.

ConflictPaused or AwaitingReconnect for many devices

Expected after point-in-time restore

For more information about post-restore device status changes, see Additional resources.

flightctl-restore exits with an error

Archive restore, database import, PKI restore, or device preparation stopped before completion

Keep application services stopped or workloads scaled to zero. Read the command output and logs. Fix the reported error (credentials, connectivity, permissions, disk space). Re-run flightctl-restore with the same archive path while services remain down. Do not start the API until the command completes successfully.

Restore command finished but logs show device or KV key errors

Device preparation is not transactional: the KV store can be cleared and database device rows updated even when later steps fail; per-device awaiting-reconnection KV keys can fail individually while the command still exits successfully

Keep services stopped. Re-run flightctl-restore with the same archive path, or without an archive if the database and PKI are already restored and you only need device preparation. Compare the device count in the completion log with your inventory. For more information about recovering from a failed or partial restore, see Additional resources.

Recovering from a failed or partial restore
Copy link

flightctl-restore does not roll back automatically. If the command fails or stops mid-run, assess what completed before you bring application services back online.

What device preparation changes

When the command reaches device preparation (with or without an archive), it typically performs these steps in order:

  1. Clears the KV store (removes cached control-plane data).
  2. Updates enrolled devices in PostgreSQL (sets the device-controller/awaitingReconnect annotation, clears last_seen, and sets summary status to AwaitingReconnect for affected devices).
  3. Updates non-approved enrollment requests for restore.
  4. Adds per-device awaiting-reconnection keys in the KV store.

If step 1 succeeds and a later step fails, the command exits with an error but the KV store can already be empty. If step 4 reports errors for individual devices, the command can still exit successfully; check the logs for Failed to add awaiting reconnection key or similar messages.

Archive-based restore can also fail earlier while importing the database dump, PKI, or configuration. In that case, the replacement host or cluster might be inconsistent until you complete a successful restore.

Recovery steps

  1. Keep Red Hat Edge Manager application services stopped on Red Hat Enterprise Linux, or keep API and worker deployments scaled to zero on Red Hat OpenShift Container Platform. Do not start the control plane for production use until restore completes successfully.
  2. Collect the flightctl-restore output and logs. Note whether the failure occurred during archive import or during device preparation.
  3. Fix the root cause (for example, database or KV connectivity, wrong DB_PASSWORD or KV_PASSWORD, port-forward dropped, or insufficient permissions).
  4. Re-run restore:
    • Full archive restore not yet complete: Run flightctl-restore again with the same archive path and credentials while services remain stopped.
    • Database and PKI already restored; only device preparation incomplete: Run flightctl-restore without an archive argument (same credentials and connectivity as your manual restore procedure) to retry device preparation.
  5. Review the completion log line for counts of devices and awaiting-reconnection keys updated. Investigate any devices that remain offline or lack AwaitingReconnect status after services are running. For more information about post-restore device status changes, see Additional resources.

If repeated restore attempts fail or the instance remains inconsistent, reinstall Red Hat Edge Manager at the target release from a clean state. Then run flightctl-restore again from a known-good archive, following your tested disaster-recovery runbook.

If problems persist, collect flightctl-backup or flightctl-restore logs, archive metadata.json, and deployment type, and open a support case with your Red Hat Edge Manager support channel.