Copy page

Model Routing

AI Gateway Fallback Model Policy

AI Gateway Fallback Model adds retry, timeout, and quota fallbacks to an existing model selection. Place it after AI Gateway Model Filtering. It preserves the primary selection and never creates one when the policy chain is misordered.

Configuration

The configuration shows how to configure the policy in the 'policies.json' document.

config/policies.json
{
  "name": "my-ai-gateway-fallback-model-inbound-policy",
  "policyType": "ai-gateway-fallback-model-inbound",
  "handler": {
    "export": "AIGatewayFallbackModelInboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "models": {
        "completions": {
          "fallback": "anthropic/claude-haiku-4-5",
          "quotaFallback": "openai/gpt-4o-mini"
        }
      },
      "fallbackTimeoutSeconds": 60
    }
  }
}

Policy Configuration

• name <string> - The name of your policy instance. This is used as a reference in your routes.
• policyType <string> - The identifier of the policy. This is used by the Zuplo UI. Value should be ai-gateway-fallback-model-inbound.
• handler.export <string> - The name of the exported type. Value should be AIGatewayFallbackModelInboundPolicy.
• handler.module <string> - The module containing the policy. Value should be $import(@zuplo/runtime).
• handler.options <object> - The options for this policy. See Policy Options below.

Policy Options

The options for this policy are specified below. All properties are optional unless specifically marked as required.

• models (required) <object> - Fallbacks grouped by AI Gateway capability.
  • completions <object> - No description available.
    • fallback <string> - The model attempted after a retryable error or timeout.
    • quotaFallback <string> - The model selected after a usage-limit-exceeded signal.
  • embeddings <object> - No description available.
    • fallback <string> - The model attempted after a retryable error or timeout.
    • quotaFallback <string> - The model selected after a usage-limit-exceeded signal.
• fallbackTimeoutSeconds <number> - How long to wait for primary-model response headers before attempting the failure fallback. The timeout does not bound a streaming response body after its headers arrive. Defaults to 60.

Using the Policy

Use this policy to add resilience to a model selection created by Model Filtering or a custom routing policy. Place it after Model Filtering:

Code
Model Filtering -> Fallback Model -> AI Gateway handler

Fallback Model never creates a primary selection. If it runs without a prior selection, it logs a warning and leaves the request unchanged.

Policy order matters. Put Fallback Model immediately after Model Filtering. Because filtering accepts the primary selection first, a fallback cannot widen the models an application may select.

Options

models must contain completions, embeddings, or both. Each configured capability must set at least one of:

• fallback: the providerName/model attempted after a retryable error or after the primary exceeds fallbackTimeoutSeconds.
• quotaFallback: the providerName/model selected after a usage-limit-exceeded signal. This path is independent of retry and timeout fallback.

fallbackTimeoutSeconds applies to every configured fallback. It defaults to 60 seconds and accepts values from 1 through 300. The timeout covers connection setup and the wait for response headers. It does not limit how long a streaming response body may continue after its headers arrive.

A fallback that names the same model as the primary selection is skipped.

Cross-provider fallback example

Code
{
  "name": "ai-gateway-fallback-model-inbound",
  "policyType": "ai-gateway-fallback-model",
  "handler": {
    "export": "AIGatewayFallbackModelInboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "models": {
        "completions": {
          "fallback": "anthropic/claude-haiku-4-5",
          "quotaFallback": "openai/gpt-4o-mini"
        },
        "embeddings": {
          "fallback": "openai/text-embedding-3-small"
        }
      },
      "fallbackTimeoutSeconds": 30
    }
  }
}

This configuration preserves the primary model selected earlier in the chain. For completions, Anthropic becomes the retry/timeout backup and OpenAI becomes the independent quota fallback. The gateway validates every reference against the live provider catalog and resolves credentials when the selection is set.

Composition behavior

Only capabilities already present in the current selection are modified. Capabilities not configured here are preserved, as are fields this policy does not override. A configured fallback replaces an existing backup and timeout; a configured quotaFallback replaces an existing quota fallback.

Retry and timeout fallback is available for translated Chat Completions and Embeddings requests. Native create requests at /v1/messages and /v1/responses use a compatible completions backup only after the primary exhausts its upstream 429 retries; they do not inherit fallbackTimeoutSeconds or transport/5xx fallback. Responses management operations do not retry a backup because stored response IDs belong to the provider that created them. Quota fallback is a separate path that the configuration executor activates when a budget is exceeded.

Write your own fallback policy

Fallback Model also uses the public routing primitives, so custom code can choose fallbacks dynamically:

• AIGatewayModels.load(context) returns the cached provider catalog, including each model's capability, status, and per-token prices (tokenRateCard).
• AIGatewayModelRouting.get(context) returns the sanitized, normalized routing selected earlier in the chain. It never returns provider credentials.
• AIGatewayModelRouting.set(context, routing) validates every target, resolves provider credentials internally, and replaces the complete stored selection.

Because set() replaces rather than partially updates routing, a custom fallback policy must read, merge, and set. It should leave the request unchanged when no earlier policy created a selection:

Code
import {
  AIGatewayModelRouting,
  AIGatewayModels,
  type ZuploContext,
  type ZuploRequest,
} from "@zuplo/runtime";

export default async function addFallback(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const routing = AIGatewayModelRouting.get(context);
  const current = routing?.completions;
  if (!routing || !current) {
    return request;
  }

  const providers = await AIGatewayModels.load(context);
  const anthropic = providers.find(
    (provider) => provider.providerName === "anthropic",
  );
  const fallback = anthropic?.models.find(
    (candidate) =>
      candidate.capability === "completions" && candidate.status === "active",
  );
  if (!fallback) {
    return request;
  }

  await AIGatewayModelRouting.set(context, {
    ...routing,
    completions: {
      ...(typeof current === "string" ? { main: current } : current),
      backup: `anthropic/${fallback.model}`,
      fallbackTimeoutSeconds: 30,
    },
  });
  return request;
}

Place this custom policy after Model Filtering, replacing Fallback Model in the inbound chain. Keeping the no-selection guard prevents a misordered custom policy from creating a primary selection and bypassing filtering.

Read more about how policies work