# AI Gateway routing and reliability > **Scope of this page** > > This page documents the Zuplo **gateway** runtime (API, AI, and MCP gateways), not the Developer Portal. > > It applies to every Zuplo hosting option: Managed Edge, Managed Dedicated, and Self-Hosted. See [Hosting options](https://zuplo.com/docs/articles/hosting-options). An app's [policy chain](./policy-chains.mdx) decides which model serves each request and what happens when that model fails or a budget runs out. Three policies cover this, and you can combine them in one chain: | You want to | Use | How it picks or changes the model | | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | Spread requests across deployments, accounts, or providers to combine rate limits | [Model Load Balancing](../policies/ai-gateway-load-balancing-inbound.mdx) | Round robin through a list, or sticky routing that keeps a user or conversation on one model | | Send simple prompts to a cheaper model and hard prompts to a stronger one | [Smart Router](../policies/ai-gateway-smart-router-inbound.mdx) | Classifies each prompt's complexity as `low`, `medium`, or `high` and routes to the model for that level | | Keep serving when the model errors, is rate limited, or is slow to respond | [Fallback Model](../policies/ai-gateway-fallback-model-inbound.mdx) `fallback` | Moves the request to a backup model after a retryable failure or timeout | | Keep serving, on a cheaper model, after a budget runs out | [Fallback Model](../policies/ai-gateway-fallback-model-inbound.mdx) `quotaFallback` | Sends the request to a quota fallback model instead of returning `429` | [Model Filtering](../policies/ai-gateway-model-filtering-inbound.mdx) is the simplest way to pick a model: the client names the model, and Model Filtering accepts it, rejects it, or supplies a default from an allow list. The policies on this page build on that selection or replace it. Request Pick the model: Model Filtering, Model Load Balancing, or Smart Router Fallback Model adds a backup and a quota fallback Budget check Chosen model Quota fallback model Backup model ## Model Load Balancing [Model Load Balancing](../policies/ai-gateway-load-balancing-inbound.mdx) spreads requests across a list of models. Use it to combine the rate limits of several deployments of one model, to serve one model from more than one provider, or to split traffic across different models. The policy picks the model for every request it handles, so the list also limits which models clients can reach. Clients still send a `model`, but any value works. Choose a `strategy`: | Strategy | How it picks a model | Use it to | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `round-robin` (default) | Cycles through the list | Spread load evenly across deployments, accounts, or providers | | `sticky` | Sends every request with the same key to the same model, on every gateway instance. Set `stickyBy` to `user`, `conversation`, or `expression` to choose the key | Keep each caller on one model, or keep each conversation on the deployment whose prompt cache already holds it | With AI Gateway API keys, `stickyBy: "user"` keeps each app on one model. To keep each end user on one model, have the app send an end-user ID in a header, set `stickyBy` to `expression`, and set `expression` to a selector such as `request.headers.get("x-end-user-id")`. Most requests also get a **retry model**: the model the strategy would pick next. With `sticky` routing, the retry model is stable for each key too. The gateway moves a request to the retry model when the chosen model fails: | When the chosen model | Chat Completions and Embeddings | Responses and Messages | | ---------------------------------------------------------------------- | ----------------------------------------------- | ----------------------------------------------- | | Returns `429` | Retries it twice, then moves to the retry model | Retries it twice, then moves to the retry model | | Returns a `5xx`, `408`, or `425` status, or the gateway can't reach it | Moves to the retry model | Fails the request | | Doesn't start responding within `fallbackTimeoutSeconds` | Moves to the retry model | Keeps waiting | Each request tries at most two models. `fallbackTimeoutSeconds` defaults to 60 seconds. ## Smart Router [Smart Router](../policies/ai-gateway-smart-router-inbound.mdx) classifies the last user prompt of each Chat Completions, Responses, and Messages request by intent and complexity, then routes the request to the model you assign to that complexity in `modelsByComplexity`: ```json { "classifier": { "app": { "appId": "$env(CLASSIFIER_APP_ID)", "model": "openai/gpt-4o-mini" } }, "modelsByComplexity": { "low": "openai/gpt-4o-mini", "medium": "openai/gpt-4o", "high": "openai/gpt-5" } } ``` The classifier runs on another app on the same gateway, in-process, so its cost shows up under that app and counts toward that app's budgets. Give the classifier app the [Internal Only](../policies/internal-only-inbound.mdx) policy instead of API Key Authentication, and keep Smart Router off its chain. Smart Router routes a request only when the classification is confident enough (`minConfidenceForRouting`, `0.5` by default) and the intent is one it knows. If the classifier fails or times out, the request continues to the model it would have used without Smart Router. To try a classifier without changing any routing, set `smartRoutingEnabled` to `false` and read the results in a later policy. ## Fallback Model [Fallback Model](../policies/ai-gateway-fallback-model-inbound.mdx) adds two independent backups to the model that an earlier policy picked. It never picks a primary model itself, so place it after the policy that does. - `fallback`: the model the gateway tries after a retryable failure (a `5xx`, `408`, `425`, or `429` status, or a network failure) or after the primary doesn't start responding within `fallbackTimeoutSeconds`. The timeout defaults to 60 seconds and accepts 1 through 300. Responses and Messages requests move to the `fallback` only when the primary keeps returning `429`. - `quotaFallback`: the model the gateway uses when an app, pool, or gateway **Block** budget runs out, instead of returning `429 Too Many Requests`. The backups can be on any provider. After Model Load Balancing, a `fallback` becomes the request's retry model in place of the list's next pick. For the setup steps and how the two backups combine, see [Fallback Models](./fallback.mdx). ## Quota fallback Budgets come from the [usage limits](./usage-limits.mdx) at the gateway, pool, and app levels, and from **Budgets and Costs** in the app's chain. When a **Block** budget runs out, requests to a provider go to the `quotaFallback` model, usually a smaller and cheaper one, so the app keeps answering while overflow traffic costs less. The quota fallback's own usage still counts toward every applicable limit. Without a `quotaFallback`, the gateway returns `429`. A cache hit doesn't use the quota fallback. When a **Block** budget runs out, cache hits return `429`. See [Successful cache responses](./usage-limits.mdx#successful-cache-responses). ## Put them in order Place the policy that picks the model first, then Fallback Model, then Budgets and Costs and Semantic Cache: ```text API Key Authentication → Model Load Balancing (or Model Filtering → Smart Router) → Fallback Model → Budgets and Costs → Semantic Cache ``` - **API Key Authentication** comes first, so sticky routing by `user` has a caller to key on. - **Model Load Balancing** doesn't need Model Filtering for the request types it has a list for, because its list already limits which models clients reach. If Model Filtering runs first, its selection stands and Model Load Balancing does nothing. - **Smart Router** goes after Model Filtering, so the allow list checks the client's model first. When Smart Router routes a request, its model wins over a Model Load Balancing pick, and the request loses the list's retry model. Place Fallback Model after Smart Router so routed requests still get its backups. - **Fallback Model** comes after the policy that picks the model. It adds nothing to a request that has no selection yet. To add guardrails to the same chain, see the [recommended starting chain](./guardrails.mdx#recommended-starting-chain). ## Example: two deployments with a budget This app spreads traffic across two Azure deployments of the same model to combine their rate limits, and moves to a cheaper model when its monthly budget runs out. Model Load Balancing: ```json { "strategy": "round-robin", "models": { "completions": ["azure-eastus/gpt-5", "azure-westus/gpt-5"] } } ``` Fallback Model, with only a quota fallback, so failed requests stay on the two deployments: ```json { "models": { "completions": { "quotaFallback": "openai/gpt-5-mini" } } } ``` Budgets and Costs blocks the app at $500 a month. Here's what happens to Chat Completions requests: | Situation | What the gateway does | | ------------------------------------------------------- | ------------------------------------------------------------------------------ | | The app is within budget | Sends each request to the next deployment in the rotation | | `azure-eastus/gpt-5` returns `429` three times in a row | Moves the request to `azure-westus/gpt-5` | | `azure-westus/gpt-5` returns `503` | Moves the request to `azure-eastus/gpt-5` | | The app reaches $500 for the month | Sends requests to `openai/gpt-5-mini` | | `openai/gpt-5-mini` returns `503` while over budget | Moves the request to the retry model from the list, so it still gets an answer | To move failed requests to a model outside the list instead, give Fallback Model a `fallback` too. It then replaces the list's retry model. ## Configure routing in the Portal Open the [Apps](https://portal.zuplo.com/+/account/project/ai/apps) tab of your AI Gateway project, select the app, and on its **Policies** tab click **Add policy**: - **Model Filtering** and **Fallback Model** have their own forms. Fallback Model's form has **Failure fallback** and **Quota fallback** fields for completions and embeddings, and a **Fallback timeout (seconds)** field. - **Model Load Balancing** and **Smart Router** open a settings form built from the policy's options, with a JSON view. For Smart Router, pick the classifier app from the list of apps in the project. Drag the entries into the order from [Put them in order](#put-them-in-order). Chain changes apply without a deploy. To give every new app in a pool the same routing, add the policies to the pool's [policy template](./policy-templates.mdx). The **Add policy** dialog lists the policies declared in your gateway's `config/policies.json`. If you connected a Git repository and a routing policy is missing from the list, see [Keep the gateway up to date](./source-control.mdx#keep-the-gateway-up-to-date). To pick models with your own logic, such as the cheapest active model, write a custom policy. See [Dynamic model routing](./cookbooks/dynamic-model-routing.mdx) and [Custom fallback logic](./cookbooks/custom-fallback.mdx). ## Related resources - [Fallback Models](./fallback.mdx): configure the backup and quota fallback - [Usage Limits](./usage-limits.mdx): set the budgets that trigger a quota fallback - [Guardrails](./guardrails.mdx): inspect prompts and responses in the same chain