Read latency, quota, and failover signals correctly

One average is not enough#

Average latency is easy to read but can hide a small group of very slow requests. P95 shows the threshold under which 95 percent of requests finish; P99 exposes the rare tail that is often most frustrating.

Read latency together with request count. A P95 based on a handful of requests is not as trustworthy as one from steady traffic, and a model producing long output naturally has a different profile from a short-response model.

Separate three kinds of quota#

• Plan quota is the user's allowance and must be enforced by the gateway.
• Rate limit bounds request speed within a short time window.
• Provider quota is the upstream connection's remaining capacity and is only an input to the router.

Clients do not need a provider's credential or detailed quota. If one connection is exhausted, the router can remove it from the candidate set and select another while preserving the public model ID.

A failover count is not automatically a failure#

A successful failover means the system recovered, but a sudden increase still deserves investigation. Compare failover count with upstream status, latency, error category, and route configuration.

Retry count also needs context. Too many retries can amplify an upstream incident, add latency, and consume quota. Policy should bound attempts, backoff, and retryable statuses.

How clients should react#

1. On 400, fix the request instead of replaying it unchanged.
2. On a public 401 or 403, check the API key and permission; do not infer an upstream secret.
3. On 429, respect Retry-After when present and apply jittered backoff.
4. On temporary 5xx errors, use a bounded retry only when the operation is safe.
5. After a stream starts, never append a replayed response to the same output.

A useful dashboard leads to an action#

A good chart is more than green or red. It should answer which model is affected, when it began, which route failed over, whether users saw errors, whether quota settled correctly, and whether the incident recovered.

Read together, latency, quota, and failover tell the story of a request lifecycle instead of acting as three disconnected widgets.