Backup, restore, and recovery drills

The Halro data directory is one consistency unit. Database, Ledger, Audit, Usage, and local Provider objects must be backed up and restored together. Never copy only halro.db or combine database, ledger, audit, and usage files from different points in time.

Keep three recovery materials separate

A recoverable backup needs at least:

the encrypted .hmbk archive;
its independent 32-byte Backup Key;
the File-mode Master Key used when the archive was created, or the corresponding Key Slot/KMS descriptor and target recovery identity.

Halro backups do not contain the Master Key, Backup Key, TLS private keys, Metrics Token, cloud identity token, or Recovery credential. Archive, Backup Key, and Master Key must not share the same volume, bucket, snapshot, or failure domain.

Failure diagnostic captures are not archived either. A retained previous_data_dir can still contain unexpired diagnostic material and must remain protected and cleaned under the original privacy-retention policy.

Create and verify a backup

Backup creation, restore, and maintenance commands that access the data directory require exclusive ownership. Stop Halro and confirm that it released the lock; never delete a lock file to bypass a failure. backup verify reads only the archive and Backup Key and can run independently while an instance is live, but it does not replace an isolated restore drill.

Create a dedicated Backup Key:

umask 077
openssl rand 32 > /secure-secrets/halro-backup.key
test "$(wc -c < /secure-secrets/halro-backup.key | tr -d ' ')" -eq 32

The archive output must be a new absolute path outside the data directory:

halro backup create \
  --config /etc/halro/config.yaml \
  --output /secure-backups/halro-2026-09-05.hmbk \
  --key-file /secure-secrets/halro-backup.key

halro backup verify \
  --file /secure-backups/halro-2026-09-05.hmbk \
  --key-file /secure-secrets/halro-backup.key

Retain the non-sensitive authenticated manifest, Backup ID, binary version, creation time, Master Key generation or Key Slot descriptor, and archive-upload result. backup verify proves archive authentication, manifest consistency, and file checksums; it does not prove that the target still controls the matching Master Key or KMS permissions.

Isolated restore drill

Run a drill in an isolated environment with a Halro version compatible with the archive. Use a dedicated configuration whose listeners remain on loopback and whose storage.data_dir points to a child of the drill volume. In File mode, use the original Master Key recorded for the archive. Never use the production data directory as the drill target.

Verify first, copy the exact returned Backup ID, and then restore:

halro backup verify \
  --file /secure-backups/halro-2026-09-05.hmbk \
  --key-file /secure-secrets/halro-backup.key

halro backup restore \
  --config /etc/halro/staging-config.yaml \
  --file /secure-backups/halro-2026-09-05.hmbk \
  --key-file /secure-secrets/halro-backup.key \
  --confirm-backup-id bkp_REPLACE_WITH_VERIFIED_ID

--confirm-backup-id is an exact confirmation for a destructive operation. Never guess it or reuse an ID from another archive. A successful restore retains the previous target directory and returns it as previous_data_dir; do not delete it before validation and the rollback-retention window complete.

Keep external traffic closed after restore and run:

halro doctor --config /etc/halro/staging-config.yaml
halro audit verify --config /etc/halro/staging-config.yaml
halro ledger verify --config /etc/halro/staging-config.yaml
halro usage verify --config /etc/halro/staging-config.yaml

Then start exactly one instance and accept /health/live, /health/ready, Admin login and MFA, Metrics, Provider connection tests, an ordinary request, a streaming request, and Usage settlement. Compare every restored enabled Gateway Key with incident and revocation records because an old backup can restore a Key that was later disabled.

If restored Deployments enter price quarantine, do not open traffic. Recheck current Provider terms and source evidence in Admin Console, then perform the authenticated restore confirmation or create a correct Price Version.

Recovery through a Key Slot

Use the Recovery Slot only in an approved recovery scenario and provide its exact configured Slot ID:

halro backup restore \
  --config /etc/halro/staging-config.yaml \
  --file /secure-backups/halro-2026-09-05.hmbk \
  --key-file /secure-secrets/halro-backup.key \
  --confirm-backup-id bkp_REPLACE_WITH_VERIFIED_ID \
  --use-recovery-slot \
  --confirm-recovery-slot slot_aws_recovery

A Recovery identity is only for offline verification and recovery, never the long-running Runtime identity. After restore, repair and independently verify a new Primary before revoking temporary Recovery/Lifecycle authority. See Master Key and Key Slot lifecycle for maturity and the complete procedure.

Production recovery gate

A drill is recoverable only when all of these conditions hold:

it used the target environment’s real key custody and permission boundary, not only backup verify;
doctor, Audit, Ledger, and Usage verification all passed;
Admin, MFA, Provider, ordinary/streaming calls, and cost settlement passed;
restored Gateway Keys and price states were reviewed;
Backup ID, version, time, operator, result, and non-sensitive evidence were recorded;
original data, image, and previous_data_dir remain available until the approved rollback window closes.