Retries, timeouts, and idempotency

Disable automatic SDK retries first

The official OpenAI and Anthropic SDKs retry some failures automatically. Behind a gateway, one visible application call can already contain multiple Provider Attempts, so an additional SDK retry can multiply cost, concurrency, and audit records. The examples use max_retries=0 or maxRetries: 0; add an explicit application policy only after classifying Halro errors.

client = OpenAI(base_url="...", api_key=..., timeout=60.0, max_retries=0)

const client = new OpenAI({ baseURL: "...", apiKey: ..., timeout: 60_000, maxRetries: 0 });

When a retry is safe

When a retry is safe

Condition	Retry?	Meaning
429 per-source / Project limit	Yes	Read Retry-After when present, then bounded backoff
429 Provider concurrency	Yes	Honor parseable Retry-After; otherwise application backoff
429 Deployment concurrency	Yes	No Retry-After; use bounded application backoff
503 provider_unavailable	Yes	No healthy Deployment currently exists
Upstream 5xx	Cautiously	May have reached Provider; treat as ambiguous and do not assume gateway fallback
403 token_guard_blocked	Not immediately	Cost threshold or temporary block; change request or wait/administer
403 model_not_allowed / 404 model_not_found	No	Configuration problem; contact administrator
Rejected 400 field	No	Request is invalid; change it
Ambiguous result	Cautiously	See next section

Indeterminate failures are not retried by the gateway

If the Provider may have accepted work but Halro cannot prove the result, a second automatic call could double-charge or duplicate side effects. Halro settles conservatively and returns an explicit failure. Record the request ID, inspect the Attempt, and let the application decide whether a new business operation is acceptable.

Resource endpoints use Idempotency-Key

Use a stable key for resource creates: Async (POST /v1/async/invocations), Files (POST /v1/files), and Batches (POST /v1/batches).

curl https://halro.example.com/v1/files \
  -H "Authorization: Bearer $HALRO_GATEWAY_KEY" \
  -H "Halro-Route: batch-files" \
  -H "Idempotency-Key: $(uuidgen)" \
  -F purpose=batch -F file=@input.jsonl

Synchronous Chat, Responses, and Messages do not use this resource-create contract. background: true Responses are the exception: Idempotency-Key is optional but strongly recommended. The same Key, body, and X-Halro-Run-ID attribution return the same record; changing any of them while reusing the Key returns 409.

Idempotency does not make an operation free: the original accepted Provider Attempt can still be billed. It prevents duplicate Halro resources and gives the caller a stable result to retrieve.

Retry-After is not always present

Source and Project rate limits can calculate a wait window and commonly return Retry-After. Deployment concurrency may not. Upstream 429 responses only carry the header when the Provider supplied a parseable value. Clients must handle the header being absent.

Timeouts

Set the client timeout longer than the expected route timeout so Halro can return its classified result. A client disconnect does not prove the Provider stopped. For streaming, apply the timeout to the interval between events rather than the complete response, or every long answer eventually times out. For long work, prefer bounded deferred Responses where supported, monitor their 24-hour TTL, and retain the final request and accounting evidence.

See Scenario 3 for combining Project/Run budgets, client retries, and Halro Attempt limits.