Copy page

Policies

AI Gateway routing and reliability

Applies to

Gatewayruntime for API, AI, and MCP gateways

Hosting

Managed EdgeManaged DedicatedSelf-Hosted

An app's policy chain decides which model serves each request and what happens when that model fails or a budget runs out. Three policies cover this, and you can combine them in one chain:

Model Filtering is the simplest way to pick a model: the client names the model, and Model Filtering accepts it, rejects it, or supplies a default from an allow list. The policies on this page build on that selection or replace it.

Model Load Balancing

Model Load Balancing spreads requests across a list of models. Use it to combine the rate limits of several deployments of one model, to serve one model from more than one provider, or to split traffic across different models. The policy picks the model for every request it handles, so the list also limits which models clients can reach. Clients still send a model, but any value works.

Choose a strategy:

With AI Gateway API keys, stickyBy: "user" keeps each app on one model. To keep each end user on one model, have the app send an end-user ID in a header, set stickyBy to expression, and set expression to a selector such as request.headers.get("x-end-user-id").

Most requests also get a retry model: the model the strategy would pick next. With sticky routing, the retry model is stable for each key too. The gateway moves a request to the retry model when the chosen model fails:

Each request tries at most two models. fallbackTimeoutSeconds defaults to 60 seconds.

Smart Router

Smart Router classifies the last user prompt of each Chat Completions, Responses, and Messages request by intent and complexity, then routes the request to the model you assign to that complexity in modelsByComplexity:

Code
{
  "classifier": {
    "app": {
      "appId": "$env(CLASSIFIER_APP_ID)",
      "model": "openai/gpt-4o-mini"
    }
  },
  "modelsByComplexity": {
    "low": "openai/gpt-4o-mini",
    "medium": "openai/gpt-4o",
    "high": "openai/gpt-5"
  }
}

The classifier runs on another app on the same gateway, in-process, so its cost shows up under that app and counts toward that app's budgets. Give the classifier app the Internal Only policy instead of API Key Authentication, and keep Smart Router off its chain.

Smart Router routes a request only when the classification is confident enough (minConfidenceForRouting, 0.5 by default) and the intent is one it knows. If the classifier fails or times out, the request continues to the model it would have used without Smart Router. To try a classifier without changing any routing, set smartRoutingEnabled to false and read the results in a later policy.

Fallback Model

Fallback Model adds two independent backups to the model that an earlier policy picked. It never picks a primary model itself, so place it after the policy that does.

• fallback: the model the gateway tries after a retryable failure (a 5xx, 408, 425, or 429 status, or a network failure) or after the primary doesn't start responding within fallbackTimeoutSeconds. The timeout defaults to 60 seconds and accepts 1 through 300. Responses and Messages requests move to the fallback only when the primary keeps returning 429.
• quotaFallback: the model the gateway uses when an app, pool, or gateway Block budget runs out, instead of returning 429 Too Many Requests.

The backups can be on any provider. After Model Load Balancing, a fallback becomes the request's retry model in place of the list's next pick. For the setup steps and how the two backups combine, see Fallback Models.

Quota fallback

Budgets come from the usage limits at the gateway, pool, and app levels, and from Budgets and Costs in the app's chain. When a Block budget runs out, requests to a provider go to the quotaFallback model, usually a smaller and cheaper one, so the app keeps answering while overflow traffic costs less. The quota fallback's own usage still counts toward every applicable limit. Without a quotaFallback, the gateway returns 429.

A cache hit doesn't use the quota fallback. When a Block budget runs out, cache hits return 429. See Successful cache responses.

Put them in order

Place the policy that picks the model first, then Fallback Model, then Budgets and Costs and Semantic Cache:

Code
API Key Authentication
  → Model Load Balancing     (or Model Filtering → Smart Router)
  → Fallback Model
  → Budgets and Costs
  → Semantic Cache

• API Key Authentication comes first, so sticky routing by user has a caller to key on.
• Model Load Balancing doesn't need Model Filtering for the request types it has a list for, because its list already limits which models clients reach. If Model Filtering runs first, its selection stands and Model Load Balancing does nothing.
• Smart Router goes after Model Filtering, so the allow list checks the client's model first. When Smart Router routes a request, its model wins over a Model Load Balancing pick, and the request loses the list's retry model. Place Fallback Model after Smart Router so routed requests still get its backups.
• Fallback Model comes after the policy that picks the model. It adds nothing to a request that has no selection yet.

To add guardrails to the same chain, see the recommended starting chain.

Example: two deployments with a budget

This app spreads traffic across two Azure deployments of the same model to combine their rate limits, and moves to a cheaper model when its monthly budget runs out.

Model Load Balancing:

Code
{
  "strategy": "round-robin",
  "models": {
    "completions": ["azure-eastus/gpt-5", "azure-westus/gpt-5"]
  }
}

Fallback Model, with only a quota fallback, so failed requests stay on the two deployments:

Code
{
  "models": {
    "completions": {
      "quotaFallback": "openai/gpt-5-mini"
    }
  }
}

Budgets and Costs blocks the app at $500 a month. Here's what happens to Chat Completions requests:

To move failed requests to a model outside the list instead, give Fallback Model a fallback too. It then replaces the list's retry model.

Configure routing in the Portal

Open the Apps tab of your AI Gateway project, select the app, and on its Policies tab click Add policy:

• Model Filtering and Fallback Model have their own forms. Fallback Model's form has Failure fallback and Quota fallback fields for completions and embeddings, and a Fallback timeout (seconds) field.
• Model Load Balancing and Smart Router open a settings form built from the policy's options, with a JSON view. For Smart Router, pick the classifier app from the list of apps in the project.

Drag the entries into the order from Put them in order. Chain changes apply without a deploy. To give every new app in a pool the same routing, add the policies to the pool's policy template.

The Add policy dialog lists the policies declared in your gateway's config/policies.json. If you connected a Git repository and a routing policy is missing from the list, see Keep the gateway up to date.

To pick models with your own logic, such as the cheapest active model, write a custom policy. See Dynamic model routing and Custom fallback logic.

Related resources

• Fallback Models: configure the backup and quota fallback
• Usage Limits: set the budgets that trigger a quota fallback
• Guardrails: inspect prompts and responses in the same chain