What happens before a request reaches an upstream model?

The request starts at a trust boundary#

The API key is authenticated at the gateway, not in the interface. The backend resolves the user, subscription, plan, scope, and restrictions before any request reaches an upstream. A UI change or custom client therefore cannot bypass policy.

The gateway then resolves the public model ID to routes the user may access. Capability validation removes candidates that do not support required features such as streaming, tool calling, or the requested protocol.

Reserve first, settle later#

Quota should be reserved before dispatch so concurrent requests cannot all observe the same remaining allowance. When the upstream finishes, the system settles the reservation with the actual outcome and usage. If no dispatch occurs, it releases the reservation according to policy.

This is stronger than a counter updated only after a response. Late updates can let a concurrent burst exceed a plan limit by a wide margin.

Where JSON and streaming diverge#

For a regular JSON response, the gateway can read a bounded payload, normalize the model ID, record usage, and return it to the client. If a candidate fails before the response is committed, policy may try the next candidate.

For Server-Sent Events, the gateway must validate the content type and bound event, buffer, and total byte sizes. Usage may arrive near the end of the stream. A terminal marker separates a complete stream from an interrupted connection.

Once data has been emitted, a disconnect has an uncertain outcome. The system must classify it conservatively instead of replaying the request and risking two generated answers.

What operational data is retained#

A useful routing trace keeps the request ID, public model, selected candidate, attempt, fallback reason, status, latency, and usage. It does not need to retain full prompts or responses by default.

Dashboards and realtime events are observational views. PostgreSQL remains the durable source, while Redis is appropriate for shared rotation state, rate limits, and short-lived counters across multiple instances.

Debug in lifecycle order#

1. Confirm the public request ID and status.
2. Check the key, entitlement, model access, and quota.
3. Inspect route candidates, health, and cooldown.
4. Compare the first attempt with any fallback attempt.
5. Confirm settlement and usage were recorded exactly once.

Following the request lifecycle separates client, policy, upstream, and metering failures instead of collapsing everything into a generic “provider error.”