Copy page

Other

Model Load Balancing Policy

Spreads AI requests across a list of models. The policy cycles through the list, or keeps requests that share a key, such as a user or a conversation, on the same model. Most requests also get a retry model from the list. A request moves there when the chosen model stays rate limited. Chat completions and embeddings requests also move there when the model returns a server error, can't be reached, or times out.

Configuration

The configuration shows how to configure the policy in the 'policies.json' document.

config/policies.json
{
  "name": "my-ai-gateway-load-balancing-inbound-policy",
  "policyType": "ai-gateway-load-balancing-inbound",
  "handler": {
    "export": "AIGatewayLoadBalancingInboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "strategy": "round-robin",
      "models": {
        "completions": [
          "azure-eastus/gpt-5",
          "azure-westus/gpt-5",
          "openai/gpt-5"
        ]
      }
    }
  }
}

Policy Configuration

• name <string> - The name of your policy instance. This is used as a reference in your routes.
• policyType <string> - The identifier of the policy. This is used by the Zuplo UI. Value should be ai-gateway-load-balancing-inbound.
• handler.export <string> - The name of the exported type. Value should be AIGatewayLoadBalancingInboundPolicy.
• handler.module <string> - The module containing the policy. Value should be $import(@zuplo/runtime).
• handler.options <object> - The options for this policy. See Policy Options below.

Policy Options

The options for this policy are specified below. All properties are optional unless specifically marked as required.

• strategy <string> - How the policy picks a model for each request. round-robin cycles through the list in order. sticky sends every request with the same key to the same model. Allowed values are round-robin, sticky. Defaults to "round-robin".
• stickyBy <string> - The key that sticky uses. user uses the authenticated caller, request.user.sub. conversation uses the conversation's system prompt and its first user message with text. expression uses the value that expression selects. Required when strategy is sticky, and not allowed with round-robin. Allowed values are user, conversation, expression.
• expression <string> - The value to key on, written like a Metering budget expression, such as request.headers.get("x-end-user-id"). Required when stickyBy is expression, and not allowed otherwise.
• models (required) <object> - The models to spread requests across, by request type. Set at least one list.
  • completions <string[]> - Models for chat completions, Responses, Messages, and System One requests, as providerName/model. From 1 through 32 models, each listed once. Provider names aren't case-sensitive, so openai/gpt-5 and OpenAI/gpt-5 are the same model. Write each model ID exactly as your provider spells it.
  • embeddings <string[]> - Models for embeddings requests, as providerName/model. From 1 through 32 models, each listed once. Provider names aren't case-sensitive. Write each model ID exactly as your provider spells it. Every entry must be the same embedding model, such as one model on several accounts.
• fallbackTimeoutSeconds <integer> - How long to wait for a model to start responding before retrying the request on another model. Applies to chat completions and embeddings. The timeout ends when the response starts, so it doesn't cut off a long streaming response. A response that doesn't stream starts only when it's complete, so the timeout covers the whole response. From 1 through 300. Defaults to 60.

Using the Policy

Use this policy to spread requests across several deployments of a model to combine their rate limits, across providers that serve the same model, or across different models. List the models, and every request the policy handles goes to one of them.

How it works

For each request, the policy:

1. Takes the list for the request's type: completions for chat completions, Responses, Messages, and System One, or embeddings for embeddings.
2. Drops models that can't serve the request's endpoint, such as an OpenAI model on /v1/messages. The policy decides this by the provider and by whether the model is Claude, not by which APIs each model serves. See Endpoints.
3. Picks a model with the strategy, plus a retry model: the model the strategy would pick next.
4. Hands both to the AI Gateway. The gateway calls the chosen model, and moves to the retry model when that call fails in a way the endpoint retries. See Failures and retries.

The gateway sends the policy's choice to the provider in place of the request's model. Clients still send a model, because most SDKs require one, but any value works. The request itself keeps the client's model, so policies that run later, such as Semantic Cache, still see that value. Because the policy picks the model for every request it handles, the list also limits which models clients can reach.

If an earlier policy already chose a model for the request, this policy leaves that choice alone. Requests of a type with no list, such as embeddings when you set only completions, pass through unchanged.

Choose a strategy

Here is how four requests could go with the example configuration, on one gateway instance:

Round robin moves each app to the next model on each of its requests. Sticky keeps app-a on azure-eastus/gpt-5, whichever gateway instance serves it.

Examples

Serve one model from several providers

Claude Sonnet is available directly from Anthropic and through Amazon Bedrock. Cycle between them to use both quotas. Use a Bedrock Mantle provider for the Bedrock entry, because the policy skips models on Bedrock Runtime providers:

Code
{
  "models": {
    "completions": [
      "anthropic/claude-sonnet-4-6",
      "bedrockmantle/anthropic.claude-sonnet-4-6"
    ]
  }
}

Keep conversations on one deployment

Providers cache the prompt prefixes they've already processed, and cached input tokens cost less and return faster. Each account or deployment has its own cache, so a conversation that moves between deployments loses the benefit:

Code
{
  "strategy": "sticky",
  "stickyBy": "conversation",
  "models": {
    "completions": [
      "azure-eastus/gpt-5",
      "azure-westus/gpt-5",
      "azure-swedencentral/gpt-5"
    ]
  }
}

This works for clients that send the whole conversation on every turn, as Chat Completions clients do. A Responses API client that continues a stored response with previous_response_id sends only the new turn, so with this list, its requests to read or continue a response fail. Use sticky routing by user or expression for those clients. See Stored responses.

Keep each customer on one model

When each of your customers calls through its own app, sticky routing by user keeps every customer on one model while customers spread across the list:

Code
{
  "strategy": "sticky",
  "stickyBy": "user",
  "models": {
    "completions": ["openai/gpt-5", "anthropic/claude-sonnet-4-6"]
  }
}

Balance embeddings

Code
{
  "models": {
    "embeddings": [
      "openai/text-embedding-3-large",
      "azure-eastus/text-embedding-3-large"
    ]
  }
}

Round robin

Round robin gives every model in the list an equal share of requests. Each gateway instance keeps a separate place in the list for each app, starting from a random point. Instances never have to coordinate, and a busy app can't skew a quiet app's rotation. Across all instances, requests spread evenly, but two requests in a row can still go to the same model.

Sticky routing

Sticky routing uses a hash, not a stored mapping, so it gives the same answer on every gateway instance. For each request, the policy scores every model by hashing the sticky key together with the model name, and picks the model with the highest score. The same key always produces the same scores, so it always gets the same model.

• The retry model is the one with the second-highest score, so it's stable too.
• Adding a model moves only the keys it now wins. Removing a model moves only the keys it was winning. Every other key stays where it was.

Requests without a key go round robin:

Stored-response requests don't go round robin when the models that store responses have more than one provider name. They fail with 400 unless a user or expression key picks their model, so under conversation they fail even with a key. See Stored responses.

Sticky by user

"stickyBy": "user" keys on request.user.sub, the identity your authentication policy sets. It works like an expression of request.user.sub.

Sticky by conversation

"stickyBy": "conversation" keys on the conversation's system prompt and its first user message with text. A client that sends the whole conversation on every turn, as Chat Completions and Messages clients do, sends the same key on every turn. On the Responses API, the system prompt is instructions plus any system or developer items before the first user message. Keeping a conversation on one model keeps it on the account whose prompt cache already holds its earlier turns.

Only text counts, including plain-text documents. Conversations that open with the same system prompt and the same first message share a model, and images and other files don't make them different. A conversation that opens with an image on its own goes round robin until a turn adds text. If your clients open every conversation the same way, send a conversation ID in a header and key on it with an expression instead.

A Responses API request that continues a stored response with previous_response_id or conversation sends only the new turn, so its key changes on every turn. See Stored responses.

The policy hashes this text in memory. It never stores or logs it.

Sticky by expression

"stickyBy": "expression" keys on one value that expression selects from the request or its context. Expressions use the same syntax as Metering budget expressions, so a value you budget on can also keep requests on one model. For example, to keep each end user on one model when your app sends an end-user ID in a header:

Code
{
  "strategy": "sticky",
  "stickyBy": "expression",
  "expression": "request.headers.get(\"x-end-user-id\")",
  "models": {
    "completions": ["azure-eastus/gpt-5", "azure-westus/gpt-5"]
  }
}

In JSON, escape the double quotes inside an expression, as the example does. Common expressions:

See Supported expressions in the Metering docs for the full list. An expression is a selector, not code, so it can't compare values, call string methods, or read the request body.

• The value follows Metering's value rules. It must be a string or a safe integer, and -0 doesn't count. The gateway converts a string to well-formed NFC Unicode, then rejects it when it's longer than 256 UTF-8 bytes or contains a control character, U+2028, or U+2029.
• Values are case-sensitive, so Acme and acme can land on different models. Two strings that normalize to the same NFC text share a model.
• When the expression resolves to nothing, to an empty string, or to a value these rules reject, such as an object, a fractional number, or an overlong string, the request goes round robin. A stored-response request fails with 400 instead when the models that store responses have more than one provider name. The request log says why.
• Per-request identifiers, such as context.requestId, aren't allowed, because a key that never repeats can't keep requests together.
• An expression outside the syntax is an options error, and the error shows where parsing stopped.

The policy hashes the value in memory. It never stores or logs it.

Failures and retries

Each request gets a retry model, except stored-response requests, /v1/messages/count_tokens requests, /v1/systemone requests, and requests that only one model in the list can serve. The AI Gateway treats the retry model like any other backup model. When it moves there depends on the endpoint:

Responses and Messages requests use the provider's native API. On those endpoints the AI Gateway moves to another model only for rate limits, as it does without this policy.

• Each request tries at most two models. The gateway retries a rate-limited model twice, after 250 ms and then 500 ms. If every attempt returns 429, the request makes six calls, three to each model, before the gateway returns 429.
• Once a model starts streaming a response, the request stays on that model.
• A request that doesn't stream gets its response only when the model finishes, so on chat completions and embeddings fallbackTimeoutSeconds limits the whole response. A model that's still working when it runs out moves the request to the retry model, which starts over with twice the time, and the request fails if that runs out too. Raise fallbackTimeoutSeconds for slow models, such as reasoning models, or stream the response. When Fallback Model runs after this policy with a fallback, raise its fallbackTimeoutSeconds too. Its timeout replaces this policy's on requests that get its fallback. See With Fallback Model.
• When only one model in the list can serve a request, this policy sets no retry model and no timeout. On /v1/responses and /v1/messages, a list that mixes providers or vendors often has only one, so list at least two models for each API your clients use. Fallback Model can also add a backup. See With Fallback Model.
• The policy doesn't remember failures between requests. While a model is down, every request that picks it tries it first: chat completions and embeddings requests then move to the retry model, and Responses and Messages requests fail. With sticky routing, the same callers pick the failing model every time.

Endpoints

If no model in the list can serve the request's endpoint, the policy returns 400. The message names the endpoint, and for an endpoint that uses a provider's own API, such as /v1/messages, the API it requires.

These rules depend on the provider, and on whether the model is Claude. The policy doesn't check which APIs each model serves. A request that picks a model on an API the model doesn't serve fails, such as a /v1/chat/completions request that picks a model that serves only the Responses API.

On /v1/responses:

• Bedrock Mantle serves the Responses API for only some of its models other than Claude. Requests that pick a Mantle model without Responses support fail. To check a model, read the APIs supported for the bedrock-mantle endpoint on its AWS model card. The AI Gateway calls that endpoint, and the bedrock-runtime endpoint's list can differ.
• OpenRouter stores no responses. When the list also has a model that does, OpenRouter gets only creates that set store to false. See Stored responses.

Stored responses

A stored response lives on the provider account that created it. Requests that read or continue one must reach that account: retrieving a response, listing its input items, deleting it, and creating a response with previous_response_id, a conversation, or an item_reference input item.

• The provider name is the part before the slash, compared without regard to case. openai/gpt-5 and openai/gpt-5-mini share one, but azure-eastus/gpt-5 and azure-westus/gpt-5 are two.
• Stored-response requests get no retry model, because no other account holds the response. Fallback Model placed after this policy still adds its fallback, so a continuation that stays rate limited moves to the fallback, and fails there if the fallback is on another account.
• A provider that stores nothing, such as OpenRouter, never gets stored-response requests. It gets a create only when the create sets store to false, or when no provider in the list stores responses, because a create stores its response unless it opts out.
• A create that moves to the retry model stores its response on the retry model's account. If that's another account, later requests for the response go to the chosen model's account and fail.
• The retry model, a fallback, or a quotaFallback is on another account when its provider name differs from the chosen model's. It's also on another account when AI Gateway Authentication passes callers' own API keys through, because the chosen model uses the caller's key and the others use your configured key.
• The 400 tells the caller only that this API can't read or continue the stored response. The fix is yours to make, so the request log carries it.
• conversation keys can't route them to one of several provider names. A retrieve sends no messages, and a request with previous_response_id sends only the new turn. With one provider name, each continuation is keyed on its new turn, so it can run on a different model than the turn before.
• Changing the list can move a caller to another model. Responses stored before the change stay on the old account.

Policy order

Place Model Load Balancing after authentication and before the policies that read or change model routing:

Code
{
  "inboundPolicyChain": [
    { "name": "ai-gateway-auth-inbound" },
    { "name": "ai-gateway-load-balancing-inbound" },
    { "name": "ai-gateway-fallback-model-inbound" },
    { "name": "ai-gateway-metering-inbound" },
    { "name": "ai-gateway-semantic-cache-inbound" }
  ]
}

This chain shows the order only. Entries without options use the options declared in config/policies.json. See Configure lists per app.

With Fallback Model

Fallback Model placed after this policy changes where failed and over-quota requests go:

• Requests this policy sends to the fallback model itself keep this policy's retry model and timeout. Requests it sends to the quotaFallback model itself get no quota fallback, so when they're over quota, the gateway returns 429. Use a quotaFallback that isn't in the list.
• When a quotaFallback call fails in a way the endpoint retries, the request moves to the retry model, so an over-quota request can still run on a model in the list. To keep over-quota requests off the list, give Fallback Model a fallback that isn't in the list. To keep failed requests inside the list, give Fallback Model only a quotaFallback.
• No model serves both /v1/responses and /v1/messages, so when your clients call both, a fallback or a quotaFallback covers at most one of them.
• Over quota, a create that continues a stored response goes to the quotaFallback, and fails there if the quotaFallback is on another account. Other creates over quota go there too, so a response they store lives on the quotaFallback's account. If that's another account, requests for the response fail once they're no longer over quota. See Stored responses.
• For embeddings, set fallback to the same embedding model as the list. With a one-model list, use that model on another provider name, because a fallback equal to the chosen model is skipped.
• Unless you've changed them, Fallback Model's declared options set openai/gpt-4o-mini as the completions fallback. A Fallback Model entry without options, like the one in the chain above, makes that model the retry model. On a gateway that can't use openai/gpt-4o-mini, such as one with no provider named openai, every request that Fallback Model adds it to fails with 500, even requests that never try a backup. Give the entry its own options.

Configure lists per app

Declare the policy in config/policies.json with a default list. An app can supply its own options in its chain entry. Entry options replace the declared options entirely:

Code
{
  "inboundPolicyChain": [
    {
      "name": "ai-gateway-load-balancing-inbound",
      "options": {
        "strategy": "sticky",
        "stickyBy": "conversation",
        "models": {
          "completions": ["azure-eastus/gpt-5", "azure-westus/gpt-5"]
        }
      }
    }
  ]
}

Unless you've changed them, the declared options are a one-model list, openai/gpt-4o-mini. A chain entry without options then sends every chat completions and Responses request to that model, and fails Messages and System One requests with 400, because that model can't serve them.

The gateway validates chain options when the policy runs. Invalid options fail requests with an error that names the field to fix. So does a model the gateway can't use, such as an unknown provider or an inactive model, on every request that picks it as the chosen or the retry model. A provider whose API key isn't on the gateway yet fails those requests with an error that says the provider isn't ready, and the request log names the provider and its environment variable.

Monitoring

AI Gateway analytics attribute each request to the model that served it. Logging plugins also receive these properties on the request's log entries after the policy picks a model:

The properties record this policy's picks. They don't change when Fallback Model replaces the retry model or Smart Router replaces the choice.

Limitations

• Each list can have up to 32 models, each listed once. The policy compares entries without regard to case, so openai/gpt-5 and OpenAI/gpt-5 count as one model. Provider names work in any case, but write each model ID exactly as your provider spells it. A model ID in another case, such as openai/GPT-5, can fail to route.
• Each request tries at most two models.
• An expression can be up to 1024 bytes and 32 property segments.

Lists that mix providers or models have limits of their own:

Read more about how policies work