Production Admin bootstrap and secret lifecycle

Version boundary

admin.setup_token_file, admin.setup_token_ttl, admin setup-token generate, idempotent admin bootstrap --if-needed, and the Kubernetes Verification Job shipped in v0.8.3 and are in v0.8.4. Releases up to and including v0.8.2 do not carry these commands or configuration keys; confirm the deployed binary with halro version before running them.

The first production administrator no longer depends on reading service logs. Halro provides an interactive secret-file path and an offline automated Bootstrap Job path, separating secret delivery from runtime and observability permissions.

Choose a bootstrap mode

Choose a bootstrap mode

Mode	Intended use	Secret delivery	Runtime command
Local start	Development and evaluation	When no file is configured, a process token appears only in the operator-controlled terminal	halro start
Remote interactive	A designated person creates the administrator in a browser	Kubernetes Secret, Vault, Secrets Manager, or CSI read-only file	Init Job, then halro serve
Offline automation	Kubernetes, GitOps, CI/CD	Password file mounted only in the one-shot Bootstrap container	Bootstrap and Verification Jobs, then halro serve

Halro never reads the Kubernetes Secret API and never creates, updates, or deletes Secrets. kubelet, CSI, or a controlled injector projects files. Application engineers do not need pods/log, pods/exec, or namespace-wide Secret access.

Interactive setup-token file

Configure and generate

admin:
  setup_token_file: /run/secrets/halro/setup-token
  setup_token_ttl: 30m

setup_token_file must be absolute. setup_token_ttl defaults to 30m, must be positive, and is capped at 24h; it bounds both generated-token lifetime and the maximum remaining lifetime accepted from a file.

Generate the file offline with the exact Halro version being deployed:

halro admin setup-token generate \
  --ttl 30m \
  --output /secure/path/setup-token

The command uses a 32-byte CSPRNG and exclusively creates a new 0600 file, refusing to overwrite an existing target. It never prints the Token to stdout, stderr, application logs, or Audit. The two-line envelope contains a 49-byte setup_ Token and the UTC absolute expiry chosen at generation time.

setup_<32-byte-CSPRNG-as-unpadded-base64url>
expires_at=<RFC3339Nano UTC>

Do not deliver it with a shell literal, command substitution, environment variable, --from-literal, ticket, chat, or Git. Let the secret manager or Kubernetes client read the file directly:

kubectl -n halro create secret generic halro-bootstrap \
  --from-file=setup-token=/secure/path/setup-token

Load and invalidation semantics

• Halro reads the file only when the database has no administrator and a Setup Token is required; it stops reading after an administrator exists.
• The file is read once at process start and is not hot-reloaded. Updating or deleting a Secret does not revoke a Token already loaded by an old process.
• Absolute expires_at never extends after restart or rescheduling. After expiry, generate a new file, rotate the Secret, and start a new Pod.
• A missing, unreadable, empty, malformed, or over-long-remaining-TTL file fails closed; Halro never falls back to generation or log delivery.
• Configuring the file explicitly requires the Token even on a loopback Admin listener.
• Setup Status and errors disclose neither source/path nor the difference between an expired and incorrect Token.
• The Token becomes invalid immediately after the first-administrator transaction commits. If automatic Session creation then fails, go to sign-in rather than attempting setup again.

The UI tells users to get it from the startup terminal or their deployment administrator’s secure channel. The setup approver needs neither production logs nor kubectl exec.

Automated offline bootstrap

The main Deployment must be absent or scaled to zero, with no other writer owning the data directory. Automation uses an absolute password-file path so it cannot block on stdin:

halro init --if-needed --config /etc/halro/config.yaml
halro admin bootstrap \
  --config /etc/halro/config.yaml \
  --username admin \
  --password-file /run/secrets/halro/admin-password \
  --if-needed \
  --operation-id install-production-20260917

--password-file and stdin share the same 1024-byte limit, password policy, and single trailing LF/CRLF handling. There is no --password <value> option, and errors disclose neither password nor path.

Idempotency and recovery boundary

A stable, non-secret operation-id identifies one installation:

• First administrator, target username, Bootstrap Intent, and Completion are persistently linked; the administrator record and Audit Intent commit in one transaction.
• A retry with the same operation ID and username recovers or confirms Audit delivery and checkpoint, then returns already_completed without changing the password.
• A different operation ID, username mismatch, existing administrator without matching Completion, or indeterminate partial state fails closed.
• A crash at database commit, Audit append, or checkpoint boundaries can recover deterministically under the same operation ID without creating a second administrator.

Automation uses exit status for success, then reads one stable result line:

Admin bootstrap result: created (operation_id=install-production-20260917)
Admin bootstrap result: already_completed (operation_id=install-production-20260917)

Both return exit code 0. Conflict, ambiguity, input, and infrastructure failures return nonzero. Never change the operation ID merely to bypass a failure.

Audit integrity verification

admin.bootstrap records source=generated|file|offline, operation ID, and target username. It excludes Token, Token digest, password, Secret name, and file path. Completion stores the exact Audit Event ID; retry and acceptance require that authenticated-chain event to match action, target, and operation ID.

After Bootstrap completes and before Runtime starts, keep the PVC offline and run:

halro doctor --config /etc/halro/config.yaml
halro audit verify --config /etc/halro/config.yaml

doctor checks Bootstrap Completion, pending Audit Intents, and later administrator changes. audit verify authenticates the Audit HMAC chain, trusted checkpoint, and the Audit Event referenced by Completion. A failure blocks Runtime and must not be hidden by editing the database or changing operation ID.

Known break-glass boundary

admin reset-password and admin reset-mfa do not yet have the same persistent Audit Intent atomic protocol as first bootstrap. Preserve the approval/incident ID in bastion or job-platform audit and do not claim that Halro itself provides an atomic audit closure for those operations.

Production Kubernetes delivery order

The official automated Job path currently supports only storage.master_key.mode: key_slots. File Master Key initialization needs a reviewed post-generation export and durable custody protocol and is not an officially supported automated Job path yet.

PVC / complete config / Secrets / default-deny NetworkPolicy
                              ↓
One-shot Bootstrap Job (init container: init --if-needed)
                              ↓
Verification Job (doctor → audit verify)
                              ↓
Delete Jobs, password Secret, and install-only GitOps declarations
                              ↓
Revoke temporary KMS Lifecycle identity
                              ↓
Start the production Deployment with halro serve

The repository provides:

• deploy/kubernetes/halro-init-job.yaml for new-PVC initialization in interactive mode;
• deploy/kubernetes/halro-bootstrap-job.yaml for automated first-administrator creation;
• deploy/kubernetes/halro-bootstrap-verify-job.yaml for offline Doctor and Audit verification;
• deploy/kubernetes/halro-bootstrap-default-deny-network-policy.yaml as the install-only default-deny baseline;
• deploy/kubernetes/halro-aws-kms.yaml as the hardened AWS KMS runtime example.

Automated example:

kubectl -n halro create secret generic halro-admin-bootstrap \
  --from-file=admin-password=/secure/path/admin-password
kubectl -n halro apply -f deploy/kubernetes/halro-bootstrap-job.yaml
kubectl -n halro wait --for=condition=complete \
  job/halro-bootstrap-install-id --timeout=15m
kubectl -n halro delete job halro-bootstrap-install-id --wait=true
kubectl -n halro apply -f deploy/kubernetes/halro-bootstrap-verify-job.yaml
kubectl -n halro wait --for=condition=complete \
  job/halro-bootstrap-verify-install-id --timeout=10m

Kubernetes security constraints

• Bootstrap/Verification Jobs and Runtime never overlap. Prefer ReadWriteOncePod, but PVC access mode does not replace orchestration ordering and the Halro data lock.
• Jobs use a dedicated ServiceAccount and temporary KMS Lifecycle permissions; Runtime retains only reviewed Primary decrypt permission.
• Project each Secret only into the container that needs it. Never mount the administrator password into init, verification, or Runtime containers.
• Run every container non-root with read-only root filesystem, dropped capabilities, seccomp, and bounded deadline/backoff.
• AWS containers set AWS_EC2_METADATA_DISABLED=true, preventing fallback to an EC2 IMDS node role after workload-identity failure.
• Native NetworkPolicy cannot express KMS FQDN policy. Add cluster-specific allows for DNS, workload identity, and the private KMS endpoint; deny all other egress.
• ServiceAccounts need no secrets/get, pods/log, or pods/exec. Govern arbitrary Pod creation, workload patch, ephemeral-container, PVC snapshot, and external-identity impersonation as equivalent secret access.

GitOps and Secret deletion

Bootstrap is install-only state, not an every-sync hook. Use an explicit pipeline stage or a one-time Application removed after success. Never combine automatic Job TTL deletion with a controller that permanently declares the same Job, or the controller will repeatedly bootstrap.

After first-administrator commit, clean up according to the carrier:

1. The native Kubernetes Secret example uses optional: true. Remove the Secret from desired state, actually delete and reschedule the Pod, and prove the initialized instance can still sign in.
2. For CSI/injectors, first deploy a revision without the volume, mount, and synchronizer; after the new Pod is Ready and login succeeds, revoke the external Secret.
3. In automated mode, delete Job/Pod and wait for PVC detach before deleting the password Secret and temporary KMS identity, then create Runtime.

“Halro stops reading after an administrator exists” and “kubelet/CSI can construct a Pod after Secret deletion” are separate dependencies and require separate acceptance.

Secret compromise, rotation, and recovery

If a Token or first-administrator password reaches a log, ticket, chat, shell history, crash artifact, tracing system, or CI output, stop automatic deployment and preserve evidence without copying the Secret again.

No administrator exists

1. Block the Admin Setup endpoint and stop every Pod that may have loaded the Token; deleting the Secret alone does not clear old process memory.
2. Revoke the old value from GitOps desired state and the external secret manager.
3. Generate a new absolute-expiry envelope with admin setup-token generate and create a new versioned Secret.
4. Start one new Pod, complete setup, and verify the admin.bootstrap Audit record.
5. Remove mount dependencies and delete the Secret using the two-phase sequence.

If an automated password leaks before Completion exists, stop the Job, rotate the password source, and recover with the same operation ID. Use offline diagnostics to determine whether the transaction committed; never use a new ID to bypass a conflict.

Administrator already exists

The Setup Token is invalid after administrator commit, but remove every copy and review Setup, Ingress, WAF/APM, Job, application, and cloud audit records. If the first password may be known, stop Runtime and perform an approved offline reset-password using a new password file. Prove old password and sessions are invalid, then inspect administrator identity, MFA, and credentials created by any suspect Session.

CLI and process safety

Secret-bearing init, admin bootstrap, admin reset-password, admin setup-token generate, doctor, and Audit verification enable core-dump/dumpable protection before touching key material and fail closed if protection cannot be established. Production must also disable core dumps and keep crash reports, heap/profiles, and diagnostic bundles from collecting Secret file contents.

At minimum, acceptance proves: logs contain no Token/password; same operation ID recovers idempotently; different IDs fail closed; Doctor and Audit verification pass; the Bootstrap identity is revoked; Runtime privilege does not expand; and the instance still signs in after Secret deletion and Pod rescheduling.