Daily operations, integrity checks, and alerts

Online checks versus offline maintenance

Online checks do not replace offline integrity verification. doctor, backup create, backup restore, Usage maintenance, Ledger verify/seal, and Audit verify require the exclusive data-directory lock, so stop Halro first. If lock acquisition fails, locate the writer; never delete the lock file or copy individual data files around it. backup verify reads only an archive and Backup Key and may run independently while the instance is online.

config check reads only the proposed file and may run before restart. Metrics credential files have their own serialized online rotation. Treat every other command as offline unless this page explicitly says otherwise.

Daily online checks

halro healthcheck --url http://127.0.0.1:8080/health/ready
curl -fsS http://127.0.0.1:8080/health/live
curl -fsS http://127.0.0.1:8080/health/ready

Gateway and Admin both expose /health/live and /health/ready; Metrics exposes /health/live and /metrics, not readiness. live=200 proves only process liveness. For ready=503, inspect the body and Admin Console → Settings → System status/Diagnostics for drain state, Accounting availability, pricing isolation, and stale activation. Provider/Deployment health is a separate business-availability signal.

halro stats --config /etc/halro/config.yaml --interval 10s works only in compatibility mode without metrics.credential_file, where the token derives from the Master Key. With a production versioned credential file, use Prometheus or Settings → About and diagnostics; current stats cannot accept the independent token file.

Daily review should cover request volume/error/latency/fallback, Provider and Deployment health, Ledger write errors, Usage queues and analytics lag, Project budgets, Token Guard rejections, unknown prices, certificate expiry, Metrics scrape, alert delivery, external Audit Anchor freshness, backup age, verification, and the latest isolated restore drill. Missing state series can mean unknown; never interpret absence as zero or healthy.

Before a configuration change

halro config check --config /etc/halro/config.yaml

Restart through the deployment manager, then check the real Gateway readiness endpoint and logs. Do not test only the Metrics listener.

Certificate contents, Metrics TLS files, log level, and log handles can reload with SIGHUP as documented in Configuration reference; all other fields require restart.

Offline doctor

Stop Halro, then run:

halro doctor --config /etc/halro/config.yaml

doctor checks configuration, permissions, exclusive ownership, schema, Master Key/Vault, Ledger, Audit, Usage manifest, timezone, disk, and references. It does not call Providers; use audited Admin tests after startup. Key Slot deployments may run halro doctor --config /etc/halro/config.yaml --no-kms for static checks only; it does not prove KMS unlock or recovery identity. Keep Halro stopped after a failure, save redacted JSON and logs, and never truncate WALs or hand-edit the database.

Usage, Ledger, and Audit

All commands below are offline:

halro usage compact --config /etc/halro/config.yaml
halro usage verify --config /etc/halro/config.yaml
halro usage rebuild-summary --config /etc/halro/config.yaml
halro usage prune --config /etc/halro/config.yaml
halro ledger verify --config /etc/halro/config.yaml
halro ledger seal --config /etc/halro/config.yaml
halro audit verify --config /etc/halro/config.yaml
halro audit verify-anchor --config /etc/halro/config.yaml --anchors /secure-audit/halro-anchors.ndjson

Usage, Ledger, and Audit
Command	Use it when	Pass boundary
`usage compact`	Immediately write new Ledger records into Usage partitions	Produces a manifest; it is not a backup
`usage verify`	Before/after export, pruning, upgrade, and restore	Manifests, records, cost, and tokens reconcile to Ledger
`usage rebuild-summary`	Derived summaries are corrupt or unavailable	Rebuilds only from verified Ledger; never rewrites accounting history
`usage prune`	Remove expired analytics partitions by `retention_days`	Does not delete Ledger; `--before YYYY-MM-DD` makes the boundary explicit
`ledger verify`	Routine integrity, upgrade, and restore acceptance	Fails closed if the chain cannot authenticate
`ledger seal`	Seal the active generation now	Does not delete history; sealed generations remain in backup and replay
`audit verify`	Validate the local Audit chain	Proves local chain integrity, not external non-repudiation
`audit verify-anchor`	Compare against anchors kept by another host	The anchor file must come from controlled off-host pull and retention

Run usage verify and confirm retention obligations before prune. Shortening the Console window does not delete Ledger, but expanding it later does not automatically repopulate the in-memory view.

Rotate Metrics credentials

Non-loopback production Metrics uses an independent credential_file and mTLS. Never reuse a Gateway Key or place the token in YAML, environment variables, command arguments, screenshots, or tickets.

umask 077
halro metrics rotate --config /etc/halro/config.yaml --overlap 10m > /secure-secrets/halro-metrics.token.next
halro metrics list --config /etc/halro/config.yaml
halro metrics verify-audit --config /etc/halro/config.yaml

Atomically deliver the new token file to Prometheus. After at least two scrape cycles with up == 1, revoke the retiring version:

halro metrics revoke --config /etc/halro/config.yaml --version OLD_VERSION

Prove the old token returns 401, rerun metrics verify-audit, and export the non-sensitive chain head to independent audit storage.

Alerts

Create a Generic JSON Webhook under Admin Console → Operations. If authentication is required, use a separately encrypted header credential such as Authorization or X-Webhook-Token, never a URL secret. The target must use HTTPS and pass Halro private-network/SSRF policy.

Run both single-endpoint and selection tests and verify receiver, retries, deduplication, and error classification. Alert bodies exclude prompts, responses, Provider credentials, Gateway Keys, raw IPs, and raw upstream errors. Monitor halro_alert_delivery_total and queue depth; keep an independent dead-man channel for failures in alert delivery itself.

At minimum alert on target/readiness, configuration_stale, Ledger write errors, Usage lag, unhealthy Provider/Deployment, fallback or capacity pressure, Metrics/Audit Anchor, certificate expiry, and alert-delivery failure.

Bootstrap audit and Secret emergencies

After production bootstrap succeeds, stop every Halro process and run these offline checks before deleting the Jobs and bootstrap Secret:

halro doctor --config /etc/halro/config.yaml
halro audit verify --config /etc/halro/config.yaml

doctor checks Bootstrap Completion, pending Audit delivery, and later administrator changes. audit verify also verifies the HMAC chain, checkpoint, and that the admin.bootstrap Audit Event referenced by Completion exists. Every retry for one installation must reuse its stable operation-id; proceed to cleanup only after created or already_completed. A different operation ID, username mismatch, or ambiguous state fails closed—never bypass it by changing IDs, deleting data, or resetting the password.

A Setup Token is ineffective after the first administrator exists, but remove every copy and review projections, pipelines, and logs. Before an administrator exists, stop every Pod that loaded the old Token, revoke the old Secret, generate a new envelope, and start a new Pod; do not hot-replace the file. Clean up CSI or Kubernetes Secrets in two phases: first deploy a workload that no longer mounts the Secret and prove the Pod can reschedule, then revoke the external object. If the automation password leaks before Completion, stop the Job, rotate its source, diagnose, and retry with the original operation-id. See Production Admin bootstrap and secret lifecycle for the complete procedure.

Administrator accounts, MFA, and emergency recovery

Create individual administrator or read_only accounts under Settings → Administrator accounts. Server-enforced read-only access is for GET-style observation; mutation, export, key issuance, and security operations require an administrator. Never delete the final administrator or share identities.

Enroll TOTP separately for every administrator under Settings → Sign-in and security. The ten recovery codes appear in full only once; store them offline. Regeneration immediately invalidates old codes. Production should use admin.mfa_policy: required and drill one recovery-code login.

If a password or every authenticator is unavailable, stop Halro and recover locally:

halro admin reset-password \
  --config /etc/halro/config.yaml \
  --username ADMIN_NAME \
  --password-file /secure-secrets/new-admin-password
halro admin reset-mfa --config /etc/halro/config.yaml --username ADMIN_NAME

Both resets invalidate current sessions. Log in immediately, reenroll MFA, store new recovery codes offline, and verify the Audit events. The password file must use restricted permissions and an absolute path; revoke it through the organization’s secret process after use. Never place passwords or TOTP seeds in arguments, environment variables, tickets, or logs.

`configuration_stale`

When Halro detects that its authoritative configuration snapshot is stale, readiness returns 503 and the data plane fails closed with 503 configuration_stale.

Confirm /health/live is still 200 and preserve the readiness response and logs.
Record stale domains, time, and reason from activation.domains in Admin system status.
Inspect halro_activation_stale, halro_activation_stale_seconds, and the first activation error.
Repair the storage, Master Key/Vault, or policy issue and wait for background reactivation.
Restore traffic only when readiness is 200, stale gauge is 0, and every domain is current.

Never bypass the gate or edit the database. If the same cause survives multiple recovery cycles, drain the instance, preserve status and logs, confirm storage and Master Key readability, then restart under change control. Restart is not a root-cause fix.

Investigate by Request ID

Applications should log the Halro Request ID. In Admin Console → Usage → Failures, filter by Project, Deployment, and Request ID; read classification and guidance before deciding whether captured content is needed.

gateway.failure_capture is disabled by default and, when enabled, stores only allowed final failures. Clicking Show reads the payload and creates an Audit event. Redacted prompts and tool arguments remain customer data, so bound record size, daily count, and retention. Policy-rejected sensitive content is not re-saved for diagnostics, and capture failure never changes the original request result.

Close the incident with Request ID, classification, affected interval, fix, and verification evidence—never copied payload text. For upgrade, restore, or key rotation, follow Backup, restore, and recovery drills.