Copy page

Model Routing

AI Gateway Model Filtering Policy

AI Gateway Model Filtering controls which providerName/model references clients may select. Use an allowList for a curated catalog with a default, or a blockList for an open catalog with explicit exclusions. Put this policy before AI Gateway Fallback Model, which adds fallback behavior only after a primary model has passed filtering.

Configuration

The configuration shows how to configure the policy in the 'policies.json' document.

Code
{
  "name": "my-ai-gateway-model-filtering-v2-inbound-policy",
  "policyType": "ai-gateway-model-filtering-v2-inbound",
  "handler": {
    "export": "AIGatewayModelFilteringV2InboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "models": {
        "completions": {
          "allowList": ["openai/gpt-4o-mini"]
        }
      }
    }
  }
}

Policy Configuration

• name <string> - The name of your policy instance. This is used as a reference in your routes.
• policyType <string> - The identifier of the policy. This is used by the Zuplo UI. Value should be ai-gateway-model-filtering-v2-inbound.
• handler.export <string> - The name of the exported type. Value should be AIGatewayModelFilteringV2InboundPolicy.
• handler.module <string> - The module containing the policy. Value should be $import(@zuplo/runtime).
• handler.options <object> - The options for this policy. See Policy Options below.

Policy Options

The options for this policy are specified below. All properties are optional unless specifically marked as required.

• models (required) <object> - Model filtering rules grouped by AI Gateway capability.
  • completions <undefined> - Rules for chat completions, Responses, and Anthropic Messages requests.
  • embeddings <undefined> - Rules for embedding requests.

Using the Policy

AI Gateway Model Filtering

Use this policy when an AI Gateway route must restrict which models clients may select. Without it, an application can use any available model from the providers configured for the Zuplo project. The policy is optional.

Choose the setup that matches the route:

• Omit Model Filtering to let each request select any available model. Every request must provide model as providerName/model.
• Use an allowList to expose a curated set of models and provide a default model.
• Use a blockList to permit available models except for specific exclusions.
• Use a custom routing policy when model selection depends on request data or application logic that an allow list or block list cannot express.

providerName is the Provider Name configured in the Zuplo Portal. The text after the first slash is the provider-specific model ID, so model IDs may contain additional slashes.

Routing without Model Filtering

When neither Model Filtering nor a custom routing policy selects a model, the AI Gateway handler reads the request body's model and uses it as the primary routing target:

Code
{
  "model": "openai/gpt-4o-mini",
  "messages": [{ "role": "user", "content": "Hello" }]
}

The handler validates that:

• model is a string in providerName/model form;
• the Provider Name is configured;
• the model is available for the route's capability;
• the Provider Assignment has usable credentials.

Missing, bare, non-string, or malformed model values receive an OpenAI-compatible 400 invalid_request_error. A validly formatted target that cannot be fulfilled is reported as a routing configuration error with a suggested fix.

When Semantic Cache is attached, a cache hit also validates request-derived routing before returning the cached response. A removed, deprecated, or otherwise unavailable target therefore follows the same routing configuration error path instead of bypassing validation with a cached response. Native route requirements are checked too, so an Anthropic Messages cache hit cannot be served by an OpenAI-backed Provider Name.

Attach Model Filtering only when the gateway must enforce model-selection rules. An attached policy must always have a valid, non-empty models configuration.

Responses management operations

These Responses API operations do not have a request body:

• GET /v1/responses/:responseId
• GET /v1/responses/:responseId/input_items
• DELETE /v1/responses/:responseId

Because they cannot supply model, they require routing to be selected before the handler runs. Configure a completions.allowList in Model Filtering so its first entry supplies the default, or use a custom inbound policy that calls AIGatewayModelRouting.set(context, { completions: "providerName/model" }). Without preselected routing, the handler returns an OpenAI-compatible 400 invalid_request_error with this configuration guidance.

Policy order

Place Model Filtering before Fallback Model in the inbound policy chain:

Code
Model Filtering -> Fallback Model -> AI Gateway handler

Model Filtering accepts or rejects the request and creates the primary model selection. Fallback Model can then enrich that allowed selection without bypassing the filter. Fallback Model does not create a primary selection by itself.

Options

models must contain completions, embeddings, or both. Each capability chooses exactly one mode:

• allowList creates a curated catalog. Only listed models are accepted, and the first entry is used when a request omits model.
• blockList leaves the catalog open except for named models. Every request must include model.

Each configured list must contain at least one entry. A capability cannot define both allowList and blockList, and unsupported fields are rejected. Every list entry is a plain providerName/model string. Matching is case-insensitive, while configured casing is preserved for the upstream request. Route-target objects and fallback fields belong in the Fallback Model policy.

Configure every capability served by a route using this policy. A route using /v1/embeddings needs embeddings; Chat Completions, Responses, and Anthropic Messages routes need completions. If the policy is attached but the active capability has no rules, the request receives a 403 response explaining which capability to add.

Allow-list example

Code
{
  "name": "ai-gateway-model-filtering-v2-inbound",
  "policyType": "ai-gateway-model-filtering-v2",
  "handler": {
    "export": "AIGatewayModelFilteringV2InboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "models": {
        "completions": {
          "allowList": ["openai/gpt-4o-mini", "anthropic/claude-haiku-4-5"]
        },
        "embeddings": {
          "allowList": ["openai/text-embedding-3-small"]
        }
      }
    }
  }
}

Block-list example

Code
{
  "models": {
    "completions": {
      "blockList": ["openai/deprecated-model", "anthropic/deprecated-model"]
    }
  }
}

In block-list mode, an unknown Provider Name is rejected even if the model is not listed. Entries whose Provider Names are absent from the live catalog are reported as warnings because they cannot match a request.

Request behavior

Situation	Result
Allow list, request omits model	The first allow-list entry is selected.
Allow list, request names a listed model	The matching configured entry is selected.
Allow list, request names an unlisted model	403 response listing the allowed models.
Block list, request omits model	400 response asking for providerName/model.
Block list, request names a blocked model	403 response.
Either mode, request uses a bare or malformed model	400 response explaining the required format.
Policy has no rules for the route capability	403 response explaining which capability to configure.
Another inbound policy already selected routing	Model Filtering leaves that selection unchanged.

Native routes also enforce the wire format. /v1/responses requires a selection served on the OpenAI API, and /v1/messages one served on the Anthropic API. A selection qualifies when its Provider Assignment is backed by that provider type, or when the provider catalog declares the selected model's dialect as that format — Bedrock Mantle serves its openai-dialect models on /v1/responses and its anthropic-dialect models on /v1/messages. Provider Names may be custom labels; validation never infers anything from the label or the model name, and a model the catalog does not declare is rejected.

For example, this embedding request is evaluated against models.embeddings:

Code
{
  "model": "openai/text-embedding-3-small",
  "input": "Search text"
}

Adding fallbacks

Declare AI Gateway Fallback Model separately and place it immediately after this policy. Its fallback handles retryable errors and timeouts; quotaFallback handles usage-limit signals independently.

Write your own routing policy

Everything Model Filtering does is built on two public primitives, so a custom inbound policy can replace it entirely. The policy's job is to choose one allowed target and store it:

• AIGatewayModels.load(context) returns the cached provider catalog, including each model's capability, status, and per-token pricing.
• AIGatewayModelRouting.set(context, routing) validates the routing, resolves provider credentials internally, and stores the selection that the AI Gateway handler consumes.

Code
import {
  AIGatewayModelRouting,
  AIGatewayModels,
  type ZuploContext,
  type ZuploRequest,
} from "@zuplo/runtime";

export default async function selectModel(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const providers = await AIGatewayModels.load(context);
  const openAI = providers.find(
    (provider) => provider.providerName === "openai",
  );
  const model = openAI?.models.find(
    (candidate) =>
      candidate.capability === "completions" && candidate.status === "active",
  );
  if (!model) {
    throw new Error("No active OpenAI completions model is available");
  }

  await AIGatewayModelRouting.set(context, {
    completions: `openai/${model.model}`,
  });
  return request;
}

Attach the module as an ordinary inbound policy instead of Model Filtering:

Code
{
  "name": "my-model-filtering-inbound",
  "policyType": "custom-code-inbound",
  "handler": {
    "export": "default",
    "module": "$import(./modules/my-model-filtering)"
  }
}

AIGatewayModelRouting.get(context) returns the sanitized, normalized routing for the current request and never returns credentials. The AI Gateway handler consumes a custom selection even when Model Filtering is not attached. If neither kind of policy creates a selection, the handler derives one from the request's required providerName/model.

Policy order determines precedence:

1. Routing selected before Model Filtering remains authoritative because Model Filtering leaves an existing selection unchanged.
2. Model Filtering creates routing when no earlier policy selected it.
3. A custom policy placed after Model Filtering may deliberately replace that selection.
4. If no policy selects routing, the handler derives it from the request.

Prefer one policy as the primary selector so the route's intent is easy to understand.

Read more about how policies work