Copy page

AI Gateway Fallback Model (v2) Policy

AI Gateway Fallback Model (v2) adds retry, timeout, and quota fallbacks to an existing model selection. Place it after AI Gateway Model Filtering (v2). It preserves the primary selection and never creates one when the policy chain is misordered.

Configuration

The configuration shows how to configure the policy in the 'policies.json' document.

Code
{
  "name": "my-ai-gateway-fallback-model-v2-inbound-policy",
  "policyType": "ai-gateway-fallback-model-v2-inbound",
  "handler": {
    "export": "AIGatewayFallbackModelV2InboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "fallbackTimeoutSeconds": 60,
      "models": {
        "completions": {},
        "embeddings": {}
      }
    }
  }
}

Policy Configuration

• name <string> - The name of your policy instance. This is used as a reference in your routes.
• policyType <string> - The identifier of the policy. This is used by the Zuplo UI. Value should be ai-gateway-fallback-model-v2-inbound.
• handler.export <string> - The name of the exported type. Value should be AIGatewayFallbackModelV2InboundPolicy.
• handler.module <string> - The module containing the policy. Value should be $import(@zuplo/runtime).
• handler.options <object> - The options for this policy. See Policy Options below.

Policy Options

The options for this policy are specified below. All properties are optional unless specifically marked as required.

• models (required) <object> - Fallbacks grouped by AI Gateway capability.
  • completions <object> - No description available.
    • fallback <string> - The model attempted after a retryable error or timeout.
    • quotaFallback <string> - The model selected after a usage-limit-exceeded signal.
  • embeddings <object> - No description available.
    • fallback <string> - The model attempted after a retryable error or timeout.
    • quotaFallback <string> - The model selected after a usage-limit-exceeded signal.
• fallbackTimeoutSeconds <number> - How long a primary model may take before the failure fallback is attempted. Defaults to 60.

Using the Policy

AI Gateway Fallback Model (v2) Policy

Use this policy to add resilience to a model selection created by Model Filtering or a custom routing policy. Place it after Model Filtering:

Code
Model Filtering -> Fallback Model -> AI Gateway handler

Fallback Model never creates a primary selection. If it runs without a prior selection, it logs a warning and leaves the request unchanged.

Options

models must contain completions, embeddings, or both. Each configured capability must set at least one of:

• fallback: the providerName/model attempted after a retryable error or after the primary exceeds fallbackTimeoutSeconds.
• quotaFallback: the providerName/model selected after a usage-limit-exceeded signal. This path is independent of retry and timeout fallback.

fallbackTimeoutSeconds applies to every configured fallback. It defaults to 60 seconds and accepts values from 1 through 300.

Cross-provider fallback example

Code
{
  "name": "ai-gateway-fallback-model-v2-inbound",
  "policyType": "ai-gateway-fallback-model-v2",
  "handler": {
    "export": "AIGatewayFallbackModelV2InboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "models": {
        "completions": {
          "fallback": "anthropic/claude-haiku-4-5",
          "quotaFallback": "openai/gpt-4o-mini"
        },
        "embeddings": {
          "fallback": "openai/text-embedding-3-small"
        }
      },
      "fallbackTimeoutSeconds": 30
    }
  }
}

This configuration preserves the primary model selected earlier in the chain. For completions, Anthropic becomes the retry/timeout backup and OpenAI becomes the independent quota fallback. The gateway validates every reference against the live provider catalog and resolves credentials when the selection is set.

Composition behavior

Only capabilities already present in the current selection are modified. Capabilities not configured here are preserved, as are fields this policy does not override. A configured fallback replaces an existing backup and timeout; a configured quotaFallback replaces an existing quota fallback.

Retry and timeout fallback is available for translated Chat Completions and Embeddings requests. Native /v1/messages and /v1/responses requests are passed through without retrying a backup. Quota fallback remains a separate, metering-driven path.

Write your own fallback policy

Fallback Model also uses the public routing primitives, so custom code can choose fallbacks dynamically:

• AIGatewayModels.load(context) returns the cached provider catalog, including each model's capability, status, and per-token pricing.
• AIGatewayModelRouting.get(context) returns the sanitized, normalized routing selected earlier in the chain. It never returns provider credentials.
• AIGatewayModelRouting.set(context, routing) validates every target, resolves provider credentials internally, and replaces the complete stored selection.

Because set() replaces rather than partially updates routing, a custom fallback policy must read, merge, and set. It should leave the request unchanged when no earlier policy created a selection:

Code
import {
  AIGatewayModelRouting,
  AIGatewayModels,
  type ZuploContext,
  type ZuploRequest,
} from "@zuplo/runtime";

export default async function addFallback(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const routing = AIGatewayModelRouting.get(context);
  const current = routing?.completions;
  if (!routing || !current) {
    return request;
  }

  const providers = await AIGatewayModels.load(context);
  const anthropic = providers.find(
    (provider) => provider.providerName === "anthropic",
  );
  const fallback = anthropic?.models.find(
    (candidate) =>
      candidate.capability === "completions" && candidate.status === "active",
  );
  if (!fallback) {
    return request;
  }

  await AIGatewayModelRouting.set(context, {
    ...routing,
    completions: {
      ...(typeof current === "string" ? { main: current } : current),
      backup: `anthropic/${fallback.model}`,
      fallbackTimeoutSeconds: 30,
    },
  });
  return request;
}

Place this custom policy after Model Filtering, replacing Fallback Model in the inbound chain. Keeping the no-selection guard prevents a misordered custom policy from creating a primary selection and bypassing filtering.

Read more about how policies work