Production checklist

Six items a production integration has to handle that a first integration can skip. Each one names what the platform does on its own and what is left to your client.

1. Retry what a retry can fix#

• Retry 408, 429, 502, 503 and 504 with exponential backoff and jitter. Routing re-runs on every attempt, so a retry often lands on a different endpoint. It is not a rerun of the same failure.
• Honour Retry-After on 429 (seconds) and the X-RateLimit-* headers generally. Pacing to the limit is cheaper than tripping it.
• Do not retry the other 4xxs. An invalid body, an oversized payload or a 402 fails identically until the request itself changes.
• Failover is already built in. During a routed request Heimdallm walks the eligible endpoints on its own, and only errors no other endpoint could fix (invalid body, oversized payload, client abort) come back without alternatives having been tried.

2. Set a timeout under the ceiling#

A request may legitimately run long, since big prompts and slow models exist. Before the answer begins a stream is bounded twice: 120 seconds of silence on any one endpoint, and ten minutes in total across every endpoint tried. Once the answer has begun only the gap between frames is bounded, at 60 seconds, so a long answer that keeps arriving is never cut by the platform and runs until the model finishes or max_tokens is reached. A buffered (non-streaming) request is bounded at ten minutes end to end. Past any of these the request fails with 504. Set the client timeout by what the product can wait, and for long generations prefer streaming with an idle timeout over a total one.

Aborting a request mid-flight is safe. Billing covers the tokens actually generated, the generation is recorded as cancelled, and its final cost is queryable afterwards with GET /generation.

3. Give every call path a fallback#

• Pass models: up to 20 fallback ids, tried in order when the primary cannot be served at all. A primary and one backup turns an endpoint outage into a slower request rather than a failed one.
• Constrain the choice deliberately: provider.only / ignore for provider trust, provider.order to pin, max_price so a market price move can never route an unbounded price to you.
• Where a capability is load-bearing (tools, structured outputs, a seed), send it explicitly. Routing selects only endpoints that declare support for every requested optional setting.

4. Handle both ways a stream ends#

• Parse per the wire contract: split on blank lines, ignore comment keep-alives, stop at [DONE], and take the final data frame for usage.
• A stream that ends without data: [DONE] was cut off. Treat it as failed, not complete.
• A chunk with an error member is terminal. The usage in it still settles: output delivered before the failure is billed, and a stream that produced nothing has its token charges waived.
• Keep the X-Generation-Id from the response headers, so any stream can be reconciled later whichever way it ended.

5. Bound spend in the request#

• Send a realistic max_tokens. The credit hold covers prompt plus the completion ceiling, so a ceiling of "whatever the model allows" makes small balances unusable and parallel requests trip over their own reservations.
• Set spend limits at the right scopes. Per key for integration-level caps, per workspace for projects, per account as the hard fence. A breached limit answers 402 with metadata.scope naming which one.
• Lead with the stable part of a prompt. On cache-capable endpoints, repeated prompt prefixes bill at the declared cached_prompt rate, and the discount shows up in usage.prompt_tokens_details.cached_tokens.
• Rate limits count requests, not tokens. Batch embeddings inputs into arrays; prefer one longer conversation turn over several tiny round-trips.

6. Log what makes a request findable#

• Log X-Generation-Id with each response, and usage.cost where the body is parsed. It is the join key to GET /generation for anything that went wrong, and the string to quote to support.
• Poll GET /key for limit headroom and GET /credits for balance; alert on downward trends, not just zeroes.
• For daily reconciliation, GET /activity returns one UTC day of requests, tokens and spend per model. Pair it with your own ledger of generation ids.
• Branch error handling on metadata.error_type, not on messages. A 402 is a state to handle, not a crash.