Streaming responses

Halro streams standard Server-Sent Events and preserves the event shape of the northbound protocol. Chat Completions, Responses, and Anthropic Messages have different event sets; clients must not parse them as one common stream format.

Once a byte is sent, the Provider cannot change

Halro may select or retry an eligible target before response bytes are committed. After the first downstream byte, switching targets would splice two model answers into one response, so no Provider retry or fallback is attempted.

Clients therefore need two failure paths:

before the first event: handle the HTTP status and stable error code;
after streaming begins: treat disconnect or malformed termination as an indeterminate partial result and decide whether the application can start a new request.

Always record the request ID. A partial stream may still have consumed Provider tokens and can still appear in accounting.

Chat Completions

Events: chat.completion.chunk, [DONE], and error.

curl -N https://halro.example.com/v1/chat/completions \
  -H "Authorization: Bearer $HALRO_GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "chat", "max_tokens": 256, "stream": true,
        "messages": [{"role": "user", "content": "Hello"}] }'

import os
from openai import OpenAI

client = OpenAI(base_url="https://halro.example.com/v1",
                api_key=os.environ["HALRO_GATEWAY_KEY"], timeout=60.0, max_retries=0)
with client.chat.completions.stream(
    model="chat", max_tokens=256,
    messages=[{"role": "user", "content": "Hello"}],
) as stream:
    for event in stream:
        if event.type == "content.delta":
            print(event.delta, end="", flush=True)

Responses

Responses has 11 event types: response.created, response.in_progress, response.output_item.added, response.content_part.added, response.output_text.delta, response.output_text.done, response.content_part.done, response.output_item.done, response.completed, response.incomplete, and error.

If you only consume text, process response.output_text.delta, but also handle response.incomplete and error. response.incomplete is a normally terminated stream whose output is unfinished, for example because it reached max_output_tokens; it is not equivalent to response.completed.

Deferred background: true requests are retrieved by polling, not by holding an SSE connection. A synchronous Response is never stored for later retrieval.

Anthropic Messages

Events: message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop, ping, and error. ping is a content-free keepalive; skip it. Usage arrives in message_delta.

Endpoints without streaming

/v1/embeddings and /v1/messages/count_tokens do not stream. Count Tokens explicitly rejects the stream field instead of ignoring it.

See Scenario 7 for client handling before and after the first event.