Master Key and Key Slot lifecycle

Every Master Key, Key Slot, and Recovery action is offline control-plane work. Stop Halro, acquire exclusive ownership of the data directory, and create a verified backup. Never place plaintext keys, ciphertext, identity tokens, or raw native errors in arguments, logs, tickets, or audit attachments. A KMS ARN is required by key rewrap --key-reference; use it only in a controlled terminal and do not copy it into ordinary logs or public evidence.

File-mode rotation

File mode applies to:

storage:
  master_key:
    mode: file

Confirm that the Master Key is outside storage.data_dir, retain the old Key, and create a new 32-byte 0600 file:

umask 077
openssl rand 32 > /secure-secrets/halro-master-next.key
test "$(wc -c < /secure-secrets/halro-master-next.key | tr -d ' ')" -eq 32

halro key rotate \
  --config /etc/halro/config.yaml \
  --new-key-file /secure-secrets/halro-master-next.key

Successful output contains only an archivable fingerprint digest, record count, and version. Do not change storage.master_key.file before the command; reconcile the current Key with command output and configuration afterward.

If the host or command stops, keep Halro stopped and retain the old Key, new Key, data directory, and logs. Retry with the same --new-key-file. Never substitute another new Key, combine data and Keys by hand, or start a second rotation.

After rotation:

run halro doctor --config /etc/halro/config.yaml;
start Halro, sign in to Admin with MFA, and send one controlled Gateway request;
verify its Usage settlement;
create and verify a post-rotation backup;
retain the old Master Key for every pre-rotation backup until that backup expires and is disposed.

Do not copy only halro.db. The database, ledger, audit, usage, configuration, and key material must represent one consistent point in time.

Key Slot permission boundary

The current key_slots implementation uses AWS KMS. Primary and Recovery should use independent customer-managed symmetric KMS Keys and differ in at least one failure domain: Key, permission/identity, account, Region, or management.

Runtime identity has only normal Primary decrypt authority.
Recovery identity is not mounted into the normal workload.
Lifecycle identity exists only during an approved offline change.
Do not use wildcard resource permissions or static cloud Access Keys in environment variables.
Primary failure never falls back automatically to Recovery; the boundary is visible and fail-closed.

Run static checks first, then the complete check under the target Runtime identity:

halro config check --config /etc/halro/config.yaml
halro doctor --config /etc/halro/config.yaml --no-kms
halro doctor --config /etc/halro/config.yaml
halro key slot status --config /etc/halro/config.yaml

--no-kms cannot prove Vault recovery. Full doctor uses Primary for a read-only unlock and leaves cloud-audit evidence.

Normal KEK rewrap

rewrap changes only one Slot’s KMS wrapping and is not a remedy for suspected compromise. Add the new Key ARN to storage.master_key.allowed_kms_keys and set storage.master_key.primary_slot or storage.master_key.recovery_slot to the new Slot ID. Keep another independently verified unlock path unchanged, run config check, and then:

halro key rewrap \
  --config /etc/halro/config.yaml \
  --purpose primary \
  --slot-id slot_aws_primary_2026q4 \
  --key-reference arn:aws:kms:REGION:ACCOUNT:key/KEY_ID

halro key slot status --config /etc/halro/config.yaml

--slot-id must exactly match the configured target Slot for that purpose or Halro fails closed. Retire the old Slot only after independent verification and approval of the recovery window and historical-backup inventory. Read current revisions from slot status, then confirm precisely:

halro key slot revoke \
  --config /etc/halro/config.yaml \
  --slot-id slot_aws_primary_2026q3 \
  --expected-descriptor-revision DESCRIPTOR_REVISION \
  --expected-slot-revision SLOT_REVISION \
  --confirm-slot-id slot_aws_primary_2026q3 \
  --reason retirement_window_completed

Remove the old allowlist entry and revoke its Grant/Policy only after revoke and recovery evidence are archived. Rewrapping this instance does not change the descriptor or KMS Key required by historical backups.

Master Key or DEK rotation

Suspected compromise of a KMS Key, Grant, Key Policy, or decrypt-capable identity requires Master Key/DEK rotation, not rewrap. Isolate the identity, preserve cloud audit evidence, and choose a stable non-sensitive operation ID:

halro key rotate \
  --config /etc/halro/config.yaml \
  --operation-id incident-2026-09-05-001

After interruption, resume only with the same operation-id. If another unfinished operation exists, stop instead of starting a second rotation or editing data files. Run doctor, Primary/Recovery drills, Admin/MFA and Gateway acceptance, and create a new backup. Every historical backup retains its old descriptor/KMS dependency and disposal date.

Recovery Slot

When Primary is unavailable, keep listeners stopped and temporarily grant the approved Recovery identity:

halro key recover \
  --config /etc/halro/config.yaml \
  --confirm-recovery-slot slot_aws_recovery

This validates Recovery and never makes it a Runtime fallback. Use a short-lived Lifecycle identity to rewrap a new Primary, revoke every temporary grant, and run full doctor plus a cold start as the normal Runtime identity. Stay failed closed if Primary is still unavailable.

Completion gates

pre-change and post-change backups were verified and at least one isolated restore drill passed;
doctor, Audit, Ledger, Usage, readiness, Admin/MFA, and a controlled request passed;
temporary Recovery/Lifecycle grants were revoked;
every historical backup records its required old Key/descriptor, retention deadline, and disposal;
only non-sensitive command results, incident IDs, and cloud-audit correlation are archived.