Documentation · Guides

Streaming

Set "stream": true on POST /chat/completions and the response becomes a text/event-stream of OpenAI-format chunks. This page is the complete wire contract: what each frame looks like, when keep-alives arrive, how the stream ends, and how a mid-stream failure is delivered. Messages and Responses also stream, using their own named events and terminal markers. /completions and /embeddings reject "stream": true with a 400.

The frames

curl -N https://api-staging.heimdallm.com/api/v1/chat/completions \
-H "Authorization: Bearer $HEIMDALL_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "model": "deepseek/deepseek-v4-flash-0731-abliterated", "stream": true,
      "messages": [{ "role": "user", "content": "Count to three." }] }'

: HEIMDALL PROCESSING

data: {"id":"gen-9f2c7e1a4b6d","object":"chat.completion.chunk","created":1755300000,"model":"deepseek/deepseek-v4-flash-0731-abliterated","provider":"ImpCC","choices":[{"index":0,"delta":{"role":"assistant","content":"One, two"},"finish_reason":null}]}

data: {"id":"gen-9f2c7e1a4b6d","object":"chat.completion.chunk","created":1755300000,"model":"deepseek/deepseek-v4-flash-0731-abliterated","provider":"ImpCC","choices":[{"index":0,"delta":{"content":", three."},"finish_reason":"stop"}]}

data: {"id":"gen-9f2c7e1a4b6d","object":"chat.completion.chunk","created":1755300000,"model":"deepseek/deepseek-v4-flash-0731-abliterated","provider":"ImpCC","choices":[],"usage":{"prompt_tokens":13,"completion_tokens":7,"total_tokens":20,"cost":0.0000041}}

data: [DONE]

In order:

Comment lines (: …) are keep-alives. One is sent about every 15 seconds while nothing else is being written — most importantly during the wait before the first token, which is exactly when idle proxies time out. Ignore them.
Data frames each carry one JSON chat.completion.chunk. Text arrives in choices[0].delta.content, tool calls in choices[0].delta.tool_calls, reasoning (where the model produces it) in choices[0].delta.reasoning. Every chunk repeats id, model and provider. The chunk that ends generation has a non-null finish_reason.
The final data frame before the sentinel has an empty choices array and the complete usage object, cost included. Successful completion sends it — no stream_options flag is needed ({"include_usage": true} is accepted for OpenAI compatibility and changes nothing).
data: [DONE] terminates the stream.

The X-Generation-Id header is on the response like on any other; X-Provider-Name is not — each chunk already names the provider.

A parser in seven lines

Split on blank lines; ignore comment lines; strip data: ; stop on [DONE]; parse the rest as JSON. The OpenAI SDKs do this for you:

stream = client.chat.completions.create(
    model="deepseek/deepseek-v4-flash-0731-abliterated",
    messages=[{"role": "user", "content": "Count to three."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

const stream = await client.chat.completions.create({
  model: 'deepseek/deepseek-v4-flash-0731-abliterated',
  messages: [{ role: 'user', content: 'Count to three.' }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
}

Parsing by hand, with fetch:

const response = await fetch(`${BASE_URL}/chat/completions`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${apiKey}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ model, messages, stream: true }),
});
if (!response.ok || !response.body) throw new Error(`HTTP ${response.status}`);

const decoder = new TextDecoder();
let buffer = '';
for await (const bytes of response.body) {
  buffer += decoder.decode(bytes, { stream: true });
  let boundary;
  while ((boundary = buffer.indexOf('\n\n')) !== -1) {
    const frame = buffer.slice(0, boundary);
    buffer = buffer.slice(boundary + 2);
    if (!frame.startsWith('data: ')) continue; // keep-alive comment
    const payload = frame.slice(6);
    if (payload === '[DONE]') break;
    const chunk = JSON.parse(payload);
    if (chunk.error) throw new Error(chunk.error.message); // in-band failure
    process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
  }
}

Errors: before and after the first byte

A stream has two failure channels, and which one you get depends on timing:

Before anything is written — an unknown model, insufficient credits, no eligible endpoint — the response is an ordinary HTTP error with a real status code and the standard error envelope. This window lasts until the first chunk, or until one keep-alive period (~15 s) elapses, whichever comes first.
After the stream has started the HTTP status is fixed at 200. A failure now arrives in-band: one final chunk carrying an error member in the same envelope shape, with finish_reason: "error", followed by data: [DONE].

A provider that fails before delivering any of the answer is not a failure you see: the request moves to the model's next endpoint and the stream carries whichever provider actually answered, named in the provider field of every chunk. An in-band error therefore means the answer had already begun when its provider failed, that no endpoint could produce one, or that the marketplace could not maintain request accounting.

data: {"id":"gen-9f2c7e1a4b6d","object":"chat.completion.chunk","created":1755300000,"model":"deepseek/deepseek-v4-flash-0731-abliterated","provider":"ImpCC","error":{"code":502,"message":"Provider disconnected","metadata":{"error_type":"provider_error"}},"choices":[{"index":0,"delta":{"content":""},"finish_reason":"error"}]}

data: [DONE]

Treat any chunk with an error member as terminal — nothing but [DONE] follows it. A response that ends without data: [DONE] was cut off (network failure, proxy restart) and should be treated as failed, not complete.

A provider that closes its stream early is one of these failures, not a quiet end. When the connection closes cleanly after part of the answer has arrived and the provider sent no finish reason, no usage report and no [DONE], what you received is a prefix and not an answer: the chunks already delivered stand, and the stream ends with an in-band 502 whose message is The provider closed the stream before the answer finished, followed by data: [DONE]. The failure counts against the provider's uptime, and billing follows the rule for any failure after the answer began (see Aborting, and what you pay). A provider that sends a finish reason or a usage report and then closes without [DONE] has finished, and its stream is a completion.

When a provider goes quiet

A stream that has stopped moving is bounded by how long it has been silent, not only by how long the whole request has run. Two minutes of silence before any of the answer arrives ends that attempt, and one minute between frames ends it once the answer has begun. Both are far beyond what a working endpoint does: the slowest first token this marketplace has measured arrived in 32 seconds, and a model that thinks at length before speaking spends that time inside the first bound.

Which side of the first token the silence falls on is what you see. Before it, the attempt is replaced — the request moves to the model's next endpoint exactly as any other pre-answer failure does, and you are told nothing. After it, part of the answer is already yours and cannot be handed to somebody else, so the failure arrives in-band as 504 with The provider stopped sending mid-answer, and billing uses only verified provider usage observations.

The silence bound does not shorten a request that is producing. With healthy request accounting, a stream that keeps arriving runs until the model finishes or max_tokens is reached. The end-to-end deadline of ten minutes covers the request up to its first answering frame, which is where failovers happen, and stops applying past it.

A provider's own keep-alive counts toward that silence: a complete SSE comment line (: keepalive), or an Anthropic-dialect ping, restarts the clock, so an engine that buffers a long step can bridge it. It is not output (no token, first answer, usage or completion), and the ten-minute total before the answer begins keeps running through it. Silence in data and comments alike ends the attempt as above.

Aborting, and what you pay

Closing the connection closes the gateway's upstream transport. This does not prove that the provider stopped executing. Billing uses verified provider observations, not a count inferred from delivered text. A missing count remains unknown; an explicitly reported zero remains zero. Known priced components can still be charged when other usage is unknown. The final usage chunk cannot be sent over a closed connection.

A stream that ends without a usage report leaves a record that says so. usage_reported is false and tokens_prompt and tokens_completion are null — absent, because nothing measured them, rather than a request that consumed nothing. Priced components computed from verified usage are still charged; the rest is not, and no count is inferred from what was delivered.

A usage report that states one count and omits another is read member by member for the same reason. The count it states is charged and published; the one it omits is left out of the usage object and recorded as absent, since a member the endpoint never stated is unknown rather than measured at none.

Use GET /generation with the X-Generation-Id to inspect recorded usage, final cost and recovery state. A pending request is not a promise that the answer was free. A finalized recovery receipt distinguishes unknown or unpriced components from measured zero and retains the original request's financial identity.

Marketplace shutdown

On shutdown the gateway withdraws readiness and refuses new requests, including requests on an existing keep-alive connection. Admitted requests can finish during the normal drain window. With the default configuration this window is 60 seconds, followed by up to 60 seconds for interruption and accounting before the application's 120-second hard ceiling. The deployment's process-manager grace is 130 seconds. Custom normal drain is the hard ceiling minus two database query deadlines, clamped at zero; database contention can still exhaust the accounting window.

A request refused while the gateway drains is a 503 with Retry-After: 5 and X-Should-Retry: true, which the official SDKs retry on their own. A stream still producing at the drain boundary receives a marketplace 503 error in-band if its transport remains usable; its message says the marketplace is restarting and to try again in 5 seconds. The Anthropic Messages dialect types it overloaded_error and the Responses dialect server_error, the codes those harnesses retry. Repeat the request: the replacement serves it within seconds. The gateway closes its upstream transport without blaming provider uptime or dispatching a replacement provider. An earlier caller, provider or ownership failure keeps its own outcome. A hard process cut can instead disconnect the stream without a final error or usage frame; graceful delivery after a hard cut is impossible. Durable usage and reservation facts remain available for once-only recovery after ownership expires and maintenance releases the stale hold. Neither interruption nor missing usage creates a measured zero or guessed charge.

Request accounting interruption

If the gateway loses ownership of an admitted request, it stops its upstream transport and returns 503 with error_type: "server". An already started stream receives this error in-band. The marketplace does not retry that provider call or continue the same request with another provider; closing the transport does not prove that the provider stopped executing.

When accounting cannot yet finalize, GET /generation exposes the retained request evidence with record_state: "unrecorded", null final cost and separate provider observations. Once recovery finalizes, it exposes the durable recovery receipt and verified charge, which may be zero while usage remains unknown. No measured-zero generation or provider earning is invented for a missing report. Reserved credits remain subject to reservation maintenance, and this marketplace interruption does not count against provider uptime.

Streaming

The frames#

A parser in seven lines#

Errors: before and after the first byte#

When a provider goes quiet#

Aborting, and what you pay#

Marketplace shutdown#