# AI Gateway Fallback Model Policy :::note{title="AI Gateway Policy"} This policy is for use with the [AI Gateway](/docs/ai-gateway/overview). See the AI Gateway documentation to learn how to configure and govern AI models with Zuplo. ::: AI Gateway Fallback Model adds retry, timeout, and quota fallbacks to an existing model selection. Place it after AI Gateway Model Filtering. It preserves the primary selection and never creates one when the policy chain is misordered. ## Configuration The configuration shows how to configure the policy in the 'policies.json' document. ```json title="config/policies.json" { "name": "my-ai-gateway-fallback-model-inbound-policy", "policyType": "ai-gateway-fallback-model-inbound", "handler": { "export": "AIGatewayFallbackModelInboundPolicy", "module": "$import(@zuplo/runtime)", "options": { "models": { "completions": { "fallback": "anthropic/claude-haiku-4-5", "quotaFallback": "openai/gpt-4o-mini" } }, "fallbackTimeoutSeconds": 60 } } } ``` ### Policy Configuration - `name` <string> - The name of your policy instance. This is used as a reference in your routes. - `policyType` <string> - The identifier of the policy. This is used by the Zuplo UI. Value should be `ai-gateway-fallback-model-inbound`. - `handler.export` <string> - The name of the exported type. Value should be `AIGatewayFallbackModelInboundPolicy`. - `handler.module` <string> - The module containing the policy. Value should be `$import(@zuplo/runtime)`. - `handler.options` <object> - The options for this policy. [See Policy Options](#policy-options) below. ### Policy Options The options for this policy are specified below. All properties are optional unless specifically marked as required. - `models` **(required)** <object> - Fallbacks grouped by AI Gateway capability. - `completions` <object> - No description available. - `fallback` <string> - The model attempted after a retryable error or timeout. - `quotaFallback` <string> - The model selected after a usage-limit-exceeded signal. - `embeddings` <object> - No description available. - `fallback` <string> - The model attempted after a retryable error or timeout. - `quotaFallback` <string> - The model selected after a usage-limit-exceeded signal. - `fallbackTimeoutSeconds` <number> - How long to wait for primary-model response headers before attempting the failure fallback. The timeout does not bound a streaming response body after its headers arrive. Defaults to `60`. ## Using the Policy Use this policy to add resilience to a model selection created by Model Filtering or a custom routing policy. Place it after Model Filtering: ```text Model Filtering -> Fallback Model -> AI Gateway handler ``` Fallback Model never creates a primary selection. If it runs without a prior selection, it logs a warning and leaves the request unchanged. > **Policy order matters.** Put Fallback Model immediately after Model > Filtering. Because filtering accepts the primary selection first, a fallback > cannot widen the models an application may select. ## Options `models` must contain `completions`, `embeddings`, or both. Each configured capability must set at least one of: - `fallback`: the `providerName/model` attempted after a retryable error or after the primary exceeds `fallbackTimeoutSeconds`. - `quotaFallback`: the `providerName/model` selected after a usage-limit-exceeded signal. This path is independent of retry and timeout fallback. `fallbackTimeoutSeconds` applies to every configured `fallback`. It defaults to 60 seconds and accepts values from 1 through 300. The timeout covers connection setup and the wait for response headers. It does not limit how long a streaming response body may continue after its headers arrive. A fallback that names the same model as the primary selection is skipped. ## Cross-provider fallback example ```json { "name": "ai-gateway-fallback-model-inbound", "policyType": "ai-gateway-fallback-model", "handler": { "export": "AIGatewayFallbackModelInboundPolicy", "module": "$import(@zuplo/runtime)", "options": { "models": { "completions": { "fallback": "anthropic/claude-haiku-4-5", "quotaFallback": "openai/gpt-4o-mini" }, "embeddings": { "fallback": "openai/text-embedding-3-small" } }, "fallbackTimeoutSeconds": 30 } } } ``` This configuration preserves the primary model selected earlier in the chain. For completions, Anthropic becomes the retry/timeout backup and OpenAI becomes the independent quota fallback. The gateway validates every reference against the live provider catalog and resolves credentials when the selection is set. ## Composition behavior Only capabilities already present in the current selection are modified. Capabilities not configured here are preserved, as are fields this policy does not override. A configured `fallback` replaces an existing backup and timeout; a configured `quotaFallback` replaces an existing quota fallback. Retry and timeout fallback is available for translated Chat Completions and Embeddings requests. Native create requests at `/v1/messages` and `/v1/responses` use a compatible completions backup only after the primary exhausts its upstream 429 retries; they do not inherit `fallbackTimeoutSeconds` or transport/5xx fallback. Responses management operations do not retry a backup because stored response IDs belong to the provider that created them. Quota fallback remains a separate, metering-driven path. ## Write your own fallback policy Fallback Model also uses the public routing primitives, so custom code can choose fallbacks dynamically: - `AIGatewayModels.load(context)` returns the cached provider catalog, including each model's capability, status, and per-token pricing. - `AIGatewayModelRouting.get(context)` returns the sanitized, normalized routing selected earlier in the chain. It never returns provider credentials. - `AIGatewayModelRouting.set(context, routing)` validates every target, resolves provider credentials internally, and replaces the complete stored selection. Because `set()` replaces rather than partially updates routing, a custom fallback policy must read, merge, and set. It should leave the request unchanged when no earlier policy created a selection: ```typescript import { AIGatewayModelRouting, AIGatewayModels, type ZuploContext, type ZuploRequest, } from "@zuplo/runtime"; export default async function addFallback( request: ZuploRequest, context: ZuploContext, ) { const routing = AIGatewayModelRouting.get(context); const current = routing?.completions; if (!routing || !current) { return request; } const providers = await AIGatewayModels.load(context); const anthropic = providers.find( (provider) => provider.providerName === "anthropic", ); const fallback = anthropic?.models.find( (candidate) => candidate.capability === "completions" && candidate.status === "active", ); if (!fallback) { return request; } await AIGatewayModelRouting.set(context, { ...routing, completions: { ...(typeof current === "string" ? { main: current } : current), backup: `anthropic/${fallback.model}`, fallbackTimeoutSeconds: 30, }, }); return request; } ``` Place this custom policy after Model Filtering, replacing Fallback Model in the inbound chain. Keeping the no-selection guard prevents a misordered custom policy from creating a primary selection and bypassing filtering. Read more about [how policies work](/articles/policies)