Zuplo
Apps

Fallback Models

Applies to
Gatewayruntime for API, AI, and MCP gateways
Hosting
Managed EdgeManaged DedicatedSelf-Hosted

Fallbacks let an app keep serving requests when its primary model fails, times out, or an applicable usage limit is exceeded, instead of returning an error to the caller. They're configured with the Fallback Model policy in the app's policy chain.

The AI Gateway offers two fallback mechanisms, each triggered by a different condition:

MechanismTriggers when…Without a fallback set…
Fallback & TimeoutThe primary fails with a retryable error (5xx, 408, 425, 429), a network failure, or a timeoutThe gateway returns the error to the caller
Quota FallbackAn app, pool, or gateway usage limit is exceededThe request is blocked with a 429

If you set both, the fallback also backs up the quota fallback. For details, see How the two fallbacks combine.

Both are configured entirely in the Zuplo Portal, and either can route to any provider—the fallback doesn't have to share the primary's provider. Fallback models are referenced like any model, as providerName/model.

Configure fallbacks

  1. Open the Apps tab of your AI Gateway project and select the app to edit.

  2. Select the Policies tab. If the chain doesn't have a Fallback Model policy yet, click Add policy and add it—placed directly after Model Filtering.

  3. Configure the policy:

    • Fallback: the model attempted after a retryable error or timeout, for example anthropic/claude-haiku-5-5.
    • Quota fallback: the model used once a usage limit is exceeded, for example openai/gpt-6-luna.
    • Fallback timeout (seconds): how long the primary call can run before the gateway fails over. The default is 60; values from 1 to 300 are accepted.

  4. Save. The change applies within about a minute.

The request timeout applies only when a fallback model is set. If no fallback is configured, the primary model call runs unbounded.

Quota fallback

For requests continuing to a provider, the quota fallback selects an alternate, usually cheaper, model when an app, pool, or gateway usage limit is exceeded, rather than blocking the request with a 429. This keeps an app available after an applicable budget, token, or request threshold is crossed, while shifting the overflow traffic to a lower-cost model.

The Fallback Model policy supplies the quota fallback selection. Add Budgets and Costs to set the app's own limits, and place it after Fallback Model. If you leave the quota fallback empty, the gateway blocks requests with a 429 once an applicable limit is exceeded.

If a Block budget is exhausted, cache hits return 429 instead of the cached answer. Quota fallback doesn't apply to cache hits. See Successful cache responses.

Point the quota fallback at a smaller, cheaper model so overflow traffic stays inexpensive while remaining available. The fallback's own usage still counts toward every applicable limit.

How the two fallbacks combine

Both fallbacks can be active on the same app. The fallback model backs up whichever model the request goes to first:

  • A request that's within quota goes to the primary model. If the primary fails with a retryable error or times out, the request moves to the fallback model.
  • A request that's over quota goes to the quota fallback model. If the quota fallback fails the same way, the request moves to the fallback model.

As a result, an over-quota request can still run on the fallback model. Set whichever fallbacks match the failure modes you want to protect against. Neither is required.

On Chat Completions and Embeddings requests, the fallback covers every failure that the Fallback & Timeout row lists. Other requests follow narrower rules:

  • Responses and Messages requests (POST /v1/responses and POST /v1/messages) move to the fallback only on a 429. The gateway first retries the primary twice, after 250 ms and then after 500 ms. The request moves only if the primary still returns 429 and the fallback model serves the same endpoint. A 5xx error, a network failure, or a timeout doesn't move the request, and the fallback timeout doesn't apply. Over quota, these requests use the quota fallback only if it serves the endpoint. Otherwise, the gateway returns 429.
  • Token counting requests (/v1/messages/count_tokens) never use the fallback. Over quota, they use the quota fallback only if it can count tokens. Otherwise, the gateway returns 429.
  • System One requests (/v1/systemone) and requests that retrieve, delete, or list the input items of a stored response never use either fallback. Over quota, the gateway returns 429.

No model serves both /v1/responses and /v1/messages. If your clients call both endpoints, each fallback model covers at most one of them.

Related resources

  • Policy Chains - How the app's policy chain executes and the recommended policy order.
  • Usage Limits - Configure the budget, token, and request limits that trigger a quota fallback.
  • Custom Providers - Add your own provider to use as a primary or fallback model.
Last modified on
Custom PoliciesUsage Limits