ArchitectureAug 12, 20267 min read

One API key, many providers without complicating the client

How a gateway keeps a stable OpenAI-compatible surface while routing, capability checks, and failover stay server-side.

Updated Aug 12, 2026

Keep the public contract small#

A client should need only three things: one Base URL, one platform API key, and one public model ID. Connection names, upstream credentials, fallback order, and cooldown state are operational details, not part of the user contract.

This keeps applications unchanged when operators switch providers or tune a route. It also creates a clear security boundary: upstream secrets never appear in the dashboard or a public response.

How a request is routed#

Before selecting a provider, the gateway checks:

The API key, subscription, and scope are valid.
The plan permits the requested public model or alias.
A candidate supports the capability required by the request.
Its connection is active, configuration-tested, and currently usable.
Quota and rate limits allow the request to continue.

Only candidates that pass every filter enter the routing policy. Round robin therefore does not mean choosing randomly from every connection; it rotates across the currently eligible set.

Failover needs a boundary#

The gateway can try another candidate after a timeout, connection error, exhausted upstream quota, or selected temporary errors such as 429, 502, 503, and 504. It should not blindly fail over every 4xx response: an invalid request still belongs back with the client that can fix it.

Streaming has a stricter boundary. Failover is safe only before the response is committed. After the first byte reaches the client, the gateway must not replay the request because that can duplicate output.

Client integration checklist#

Discover public model IDs instead of hard-coding a provider account.
Set a bounded client timeout that still covers normal processing time.
Retry only when the error and operation are safe to retry.
Log request ID, status, and latency; never log API keys or sensitive content.
Check the status page when error rates rise across unrelated requests.

A good gateway does not erase distributed-system complexity. It contains that complexity in one server layer that can be observed, tested, and operated consistently.

Back to all articles Đọc bản tiếng Việt

Keep the public contract small#

This keeps applications unchanged when operators switch providers or tune a route. It also creates a clear security boundary: upstream secrets never appear in the dashboard or a public response.

How a request is routed#

Before selecting a provider, the gateway checks:

The API key, subscription, and scope are valid.

The plan permits the requested public model or alias.

A candidate supports the capability required by the request.

Its connection is active, configuration-tested, and currently usable.

Quota and rate limits allow the request to continue.

Only candidates that pass every filter enter the routing policy. Round robin therefore does not mean choosing randomly from every connection; it rotates across the currently eligible set.

Failover needs a boundary#

Client integration checklist#

Discover public model IDs instead of hard-coding a provider account.

Set a bounded client timeout that still covers normal processing time.

Retry only when the error and operation are safe to retry.

Log request ID, status, and latency; never log API keys or sensitive content.

Check the status page when error rates rise across unrelated requests.

A good gateway does not erase distributed-system complexity. It contains that complexity in one server layer that can be observed, tested, and operated consistently.