Route suspensions, fallback, and recovery

v0.9.0 replaces circuit_breaker with a unified route admission gate. It separates connection/Deployment unavailability from explicit upstream refusals and binds suspensions to the relevant Credential, Credential + model, or Deployment. Multiple Deployments sharing a dead credential do not each need to rediscover it.

Configuration and attempt budgets

Merge this fragment into a complete configuration; it is not a standalone file:

routing:
  availability_failures: 5
  suspend_for: 30s
  max_suspend_for: 5m
  probe_requests: 1
gateway:
  max_total_attempts: 4
retry:
  max_attempts_per_target: 1

retry.max_attempts_per_target includes the first attempt. If each candidate spends its entire allowance, the request reaches at most ceil(max_total_attempts / max_attempts_per_target) candidates: two at the old 3 / 2 settings, four at the default 4 / 1. Skipping a target before admission spends no attempt, and route_total_timeout still bounds the complete request. Existing files do not adopt new defaults automatically.

Suspension reasons and recovery

Suspension reasons and recovery

Reason	Default behavior	Operator action
invalid_credential, subscription_inactive	One explicit refusal suspends the Credential indefinitely	Repair and re-save the credential to change its revision; restarting is not a fix
subscription_quota_exhausted	Suspend Credential + model for 15 minutes initially, doubling on failed probes up to 6 hours	Check quota, billing, and product; honor upstream Retry-After when present
rate_limited	Suspend Credential + model for 1 second initially, up to 1 minute	Reduce pressure and await a probe
entitlement_verification_unavailable	Suspend Credential for 30 seconds initially, up to 5 minutes	Inspect entitlement verification
Unclassified availability failure	Suspend after routing.availability_failures consecutive failures, using routing.* windows	Inspect service status, connections, and timeout configuration

Only reasons established from the response by the adapter enter the corresponding policy. Do not interpret every 402/403 as exhausted quota. Credential suspensions have no Retry-After. Expiring a timed suspension only admits limited probes, not guaranteed recovery. When all candidates are suspended, callers receive 503 all_candidates_suspended without upstream account or credential identities.

Fallback boundaries

Chat, portable Responses/Messages, and Embeddings can walk to the next candidate before sending content to the client when a failure is explicitly retryable or causes a suspension. Ambiguous outcomes do not automatically retry or fall back; a stream cannot switch targets after emitting bytes. Native Anthropic Messages / Count Tokens do not inherit portable fallback. Media, Files, Batches, and Async need a single target; multiple candidates return 409 ambiguous_resource_route.

Suspension therefore does not mean every 5xx triggers failover, and idempotency does not guarantee delivery of an answer. See retries and idempotency for application policy.

Inspecting and clearing suspensions

Use the Console Providers / Routes suspension panels for reason, scope, credential revision, and clearable. The Admin API exposes GET /admin/api/v1/route-suspensions; clear an eligible row with DELETE /admin/api/v1/route-suspensions/{scope_id}. Use the returned opaque scope_id, never a constructed value. Clearing commits with the route_suspension.clear audit action. It does not repair the upstream cause, and another refusal can suspend the scope again.

Offline commands require stopping Halro and acquiring its data-directory lock:

halro route suspensions --config /etc/halro/config.yaml
halro route clear-suspension --config /etc/halro/config.yaml --scope-id RETURNED_SCOPE_ID

Replace RETURNED_SCOPE_ID with the exact listing value. The offline list shows persisted suspensions only. Indefinite credential suspensions and timed windows longer than five minutes are durable. Short availability suspensions and probe state do not survive as records across restarts. Replacing a bad credential remains the preferred remedy.

Advisor and monitoring

Console Settings/Diagnostics shows deterministic findings; offline halro doctor --config ... reports the same configuration comparisons in findings. ok means checked, warn identifies a problematic combination, and unknown means runtime evidence was unavailable. Rules compare response-header and total timeouts, per-target attempt budgets, fallback fan-out, suspensions, and runtime limits without calling a model.

Watch halro_route_suspended, halro_route_suspension_transitions_total, halro_route_probe_admitted_total, and halro_provider_failure_reason_total; configure HalroCredentialUnusable and HalroProviderQuotaExhausted. Look up identities through Admin panels/API, not Metrics labels or caller-facing errors.