Upgrade v0.8.5 → v0.9.0 and the v0.9.1 patch

The v0.8.5 → v0.9.0 upgrade changes the configuration schema, moves accounting and usage history to bounded memory and disk indexes, introduces reason-aware route suspensions, and adds public-alias discovery and Chat/Embeddings idempotency. This documentation is verified against v0.9.1. Upgrading directly from v0.8.5 to v0.9.1 requires the same migration.

Back up first: no in-place downgrade

The first start moves halro.db from metadata schema 38 to 41. v0.8.5 cannot open the migrated directory. Rollback requires the complete pre-upgrade backup, old configuration, old binary, and corresponding Master Key. Replacing only the executable is insufficient. Configuration version: 2 is separate from both the metadata schema and the product version.

Upgrade steps

1. Check halro version, preserve the v0.8.5 binary and configuration, stop the service, and confirm that it released the data-directory lock.
2. Create and verify a backup with v0.8.5. Follow backup and restore and store the .hmbk, Backup Key, and Master Key/KMS recovery material separately.
3. Verify the new download’s signature and checksums. Here ./halro means the new binary; substitute the actual instance configuration path.

./halro version
./halro config migrate --config /etc/halro/config.yaml
./halro config migrate --config /etc/halro/config.yaml --write

The first migration command previews without writing. --write applies the edit and preserves /etc/halro/config.yaml.before-migrate. It advances version: 1 to 2 and moves retired fields to their replacements. It preserves values you explicitly set and never runs automatically on startup. An old field whose meaning changed is refused and requires an explicit operator decision.

4. Review Ledger sealing and attempt budgets. An older file commonly retains disabled sealing, an 8 GiB threshold, and a 3 / 2 attempt budget. Migration preserves them. Choose appropriate values from the table below, then validate:

./halro config check --config /etc/halro/config.yaml

5. Start one new instance and inspect /health/live, /health/ready, logs, and Admin system status. The first start rebuilds accounting state from the Ledger, then builds usage history in the background. Before this start, the new doctor reports the schema 38/41 mismatch because it is read-only and does not migrate. Complete the first start before stopping for offline diagnostics.
6. During the maintenance window, stop the service and run the new halro doctor, halro ledger verify, halro audit verify, and halro usage verify, all with the same --config, then restart. Wait for the Usage index to catch up: readiness alone does not prove history is complete. Follow the instance acceptance procedure for requests, streaming, and cost settlement. Live Provider validation makes calls and may incur fees.

Parameter changes

Parameter changes

Parameter	v0.8.5	v0.9.0 / v0.9.1
version	1	2; migrate the old shape first
circuit_breaker.consecutive_failures	Retired field	routing.availability_failures, default 5
circuit_breaker.open_duration	Retired field	routing.suspend_for, default 30s
circuit_breaker.half_open_max_requests	Retired field	routing.probe_requests, default 1
routing.max_suspend_for	Absent	New; default 5m, no shorter than suspend_for
gateway.max_total_attempts	Default 3	Default 4
retry.max_attempts_per_target	Default 2	Default 1, including the first attempt
ledger.seal.enabled	Default false	Default true; explicitly edit old files
ledger.seal.max_active_bytes	Default 8589934592 (8 GiB)	Default 268435456 (256 MiB)
ledger.max_hot_state_bytes	Absent	New; default 134217728 (128 MiB), range 16 MiB–4 GiB
usage.console_window_days	In-memory history window	Default list date range only; default 30, minimum 7; shortening deletes nothing
provider_subscriptions.anthropic_claude	Absent	New; default false; explicitly enables the Claude subscription offering

The retired providers section is also refused: connection settings belong to Admin-managed Credentials. The older elevation_window cannot be inherited as admin.reauth_elevation_window because its authorization coverage changed. See the configuration reference for parameters and reload boundaries.

Feature and request changes

Feature and request changes

Change	Application or operator action
Model discovery (experimental)	GET /v1/models and GET /v1/models/{id} return public aliases visible to the Project; the Key needs both inference and discovery
Chat / Embeddings idempotency	Idempotency-Key prevents duplicate upstream execution without replaying answers; repeats return 409 and records expire after 24 hours
Route admission and suspensions	Recognized credential/quota refusals can trigger fallback; long suspensions survive restarts and support audited clearing
Fixed local-request rate limit	Discovery and resource endpoints have a 600/minute per-Key and 5,000/minute per-Project ceiling; Project RPM=0 does not disable it
Provider product isolation	Kimi Code and Claude subscriptions use separate Offerings, Profiles, and Credentials; sharing a protocol does not share balances
Admin Advisor	Online diagnostics and offline doctor check timeouts, attempt budgets, fallback fan-out, and suspensions; unavailable offline runtime input is unknown
Console operations	Routes group by copyable public alias and open a prefilled workbench; calls show cost and Project, and Providers show refusal reasons and credential revisions

Existing Keys do not automatically gain discovery. CLI halro key create still mints an inference-only Key. The first-run bootstrap Key explicitly includes discovery. Operators choose scopes in the Key dialog or Admin API.

See model discovery, retries and idempotency, route suspensions and recovery, and Provider offerings and subscription boundaries.

Storage, memory, and monitoring

Settled accounting state no longer remains in memory indefinitely. Startup can use an authenticated snapshot bound to the Ledger chain head; a mismatch triggers a WAL rebuild. Finished Work Units, Runs, and Outcomes move to data/history/archive.db; usage detail moves to data/history/history.db. Both indexes are derivatives that can be rebuilt from the authoritative Ledger and Governance journal.

ledger.max_hot_state_bytes bounds the estimated accounting hot state. At capacity, new requests, Work Units, and Runs receive 503 accounting_capacity_exhausted with Retry-After; admitted work continues to settle. A background Response rejected at this boundary returns to its queue. This is not a process memory limit: snapshot writing reached about 6.2 times the state size in release measurements, requiring roughly 800 MiB of heap headroom at the default, plus the rest of the process and container overhead.

Disk requirements increase. Release measurements put history.db at about 5.75 KB per request; actual size depends on workload. History rows currently do not expire automatically. usage.retention_days bounds only analytics export partitions, not the Ledger or history.db. Shortening the console window does not reclaim disk.

Update monitoring:

• Replace halro_usage_window_* and halro_usage_checkpoint_* with halro_usage_history_* for build state, watermark, pending records, and failures. Hourly reconciliation checks accounting days older than 48 hours with no open lease against the Ledger.
• Add hot-state capacity, route suspension, and upstream-refusal signals. New alerts cover Project/Run budget refusals, missing Provider spend, and Admin Audit intent backlog and delivery failures.
• Streaming first-response-byte time is separate from total latency. Explicitly set HALRO_PPROF_LISTEN=127.0.0.1:6060 to enable local Go profiles; it is off by default and rejects non-loopback listeners.

New failures include all_candidates_suspended (503), accounting_capacity_exhausted (503), and idempotency 409s. Upstream response-header timeouts now return 504 provider_timeout. Invalid-credential suspensions have no Retry-After and need operator repair. See error codes and daily operations.

Other behavior fixes

Slow metadata writes and concurrent price selections no longer falsely quarantine a Deployment as a clock rollback; actual clock anomalies still fail closed. An existing wall_clock_* pricing quarantine is not cleared by upgrading. This release has no administrative action to lift it, so the Deployment must be replaced; restore confirmation handles only restored_* quarantines.

When a background Response is rejected at Project rate/concurrency or hot-state capacity before the Ledger records admission and returns to its queue, its Token Guard admission and corresponding TPM charge are withdrawn. Repeated dequeues therefore do not repeatedly spend allowance. Failures that reached upstream retain their conservative settlement semantics.

v0.9.1 patch boundary

v0.9.1 does not migrate configuration or disk formats again. During hourly maintenance it reopens history.db and archive.db when neither is occupied by a read/write transaction, reclaiming accumulated bbolt freelist memory. A failed reopen is logged, history writes pause, and reopening retries every minute. Once successful, history catches up from the Ledger. This patch adds neither history retention nor a general process-memory ceiling.

From v0.9.0, follow normal backup and verification. From v0.8.5, complete this page’s migration. Release evidence: v0.9.0 changelog and v0.9.1 release notes.