Copy page

Model Routing

AI Gateway Model Filtering Policy

AI Gateway Model Filtering controls which providerName/model references clients may select. Use an allowList for a curated catalog with a default, or a blockList for an open catalog with explicit exclusions. Put this policy before AI Gateway Fallback Model, which adds fallback behavior only after a primary model has passed filtering.

Configuration

The configuration shows how to configure the policy in the 'policies.json' document.

Code
{
  "name": "my-ai-gateway-model-filtering-v2-inbound-policy",
  "policyType": "ai-gateway-model-filtering-v2-inbound",
  "handler": {
    "export": "AIGatewayModelFilteringV2InboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "models": {
        "completions": {
          "allowList": ["openai/gpt-4o-mini"]
        }
      }
    }
  }
}

Policy Configuration

• name <string> - The name of your policy instance. This is used as a reference in your routes.
• policyType <string> - The identifier of the policy. This is used by the Zuplo UI. Value should be ai-gateway-model-filtering-v2-inbound.
• handler.export <string> - The name of the exported type. Value should be AIGatewayModelFilteringV2InboundPolicy.
• handler.module <string> - The module containing the policy. Value should be $import(@zuplo/runtime).
• handler.options <object> - The options for this policy. See Policy Options below.

Policy Options

The options for this policy are specified below. All properties are optional unless specifically marked as required.

• models (required) <object> - Model filtering rules grouped by AI Gateway capability.
  • completions <undefined> - Rules for chat completions, Responses, and Anthropic Messages requests.
  • embeddings <undefined> - Rules for embedding requests.

Using the Policy

Use this policy when an AI Gateway route must restrict which models clients may select. Without it, an application can use any available model from the providers configured for the Zuplo project. The policy is optional.

Choose the setup that matches the route:

• Omit Model Filtering to let each request select any available model. Every request must provide model as providerName/model.
• Use an allowList to expose a curated set of models and provide a default model.
• Use a blockList to permit available models except for specific exclusions.
• Use a custom routing policy when model selection depends on request data or application logic that an allow list or block list cannot express.

providerName is the Provider Name configured in the Zuplo Portal. The text after the first slash is the provider-specific model ID, so model IDs may contain additional slashes.

Routing without Model Filtering

When neither Model Filtering nor a custom routing policy selects a model, the AI Gateway handler reads the request body's model and uses it as the primary routing target:

Code
{
  "model": "openai/gpt-4o-mini",
  "messages": [{ "role": "user", "content": "Hello" }]
}

The handler validates that:

• model is a string in providerName/model form;
• the Provider Name is configured;
• the model is available for the route's capability;
• the Provider Assignment has usable credentials.

Missing, bare, non-string, or malformed model values receive an OpenAI-compatible 400 invalid_request_error. A validly formatted target that cannot be fulfilled is reported as a routing configuration error with a suggested fix.

When Semantic Cache is attached, a cache hit also validates request-derived routing before returning the cached response. A removed, deprecated, or otherwise unavailable target therefore follows the same routing configuration error path instead of bypassing validation with a cached response. Native route requirements are checked too, so an Anthropic Messages cache hit cannot be served by an OpenAI-backed Provider Name.

Attach Model Filtering only when the gateway must enforce model-selection rules. An attached policy must always have a valid, non-empty models configuration.

Responses management operations

These Responses API operations do not have a request body:

• GET /v1/responses/:responseId
• GET /v1/responses/:responseId/input_items
• DELETE /v1/responses/:responseId

Because they cannot supply model, they require routing to be selected before the handler runs. Configure a completions.allowList in Model Filtering so its first entry supplies the default, or use a custom inbound policy that calls AIGatewayModelRouting.set(context, { completions: "providerName/model" }). Without preselected routing, the handler returns an OpenAI-compatible 400 invalid_request_error with this configuration guidance.

Policy order

Place Model Filtering before Fallback Model in the inbound policy chain:

Code
Model Filtering -> Fallback Model -> AI Gateway handler

Model Filtering accepts or rejects the request and creates the primary model selection. Fallback Model can then enrich that allowed selection without bypassing the filter. Fallback Model does not create a primary selection by itself.

Options

models must contain completions, embeddings, or both. Each capability chooses exactly one mode:

• allowList creates a curated catalog. Only listed models are accepted, and the first entry is used when a request omits model.
• blockList leaves the catalog open except for named models. Every request must include model.

Each configured list must contain at least one entry. A capability cannot define both allowList and blockList, and unsupported fields are rejected. Every list entry is a plain providerName/model string. Matching is case-insensitive, while configured casing is preserved for the upstream request. Route-target objects and fallback fields belong in the Fallback Model policy.

Configure every capability served by a route using this policy. A route using /v1/embeddings needs embeddings; Chat Completions, Responses, and Anthropic Messages routes need completions. If the policy is attached but the active capability has no rules, the request receives a 403 response explaining which capability to add.

Allow-list example

Code
{
  "name": "ai-gateway-model-filtering-v2-inbound",
  "policyType": "ai-gateway-model-filtering-v2",
  "handler": {
    "export": "AIGatewayModelFilteringV2InboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "models": {
        "completions": {
          "allowList": ["openai/gpt-4o-mini", "anthropic/claude-haiku-4-5"]
        },
        "embeddings": {
          "allowList": ["openai/text-embedding-3-small"]
        }
      }
    }
  }
}

Block-list example

Code
{
  "models": {
    "completions": {
      "blockList": ["openai/deprecated-model", "anthropic/deprecated-model"]
    }
  }
}

In block-list mode, an unknown Provider Name is rejected even if the model is not listed. Entries whose Provider Names are absent from the live catalog are reported as warnings because they cannot match a request.

Request behavior

Situation	Result
Allow list, request omits model	The first allow-list entry is selected.
Allow list, request names a listed model	The matching configured entry is selected.
Allow list, request names an unlisted model	403 response listing the allowed models.
Block list, request omits model	400 response asking for providerName/model.
Block list, request names a blocked model	403 response.
Either mode, request uses a bare or malformed model	400 response explaining the required format.
Policy has no rules for the route capability	403 response explaining which capability to configure.
Another inbound policy already selected routing	Model Filtering leaves that selection unchanged.

Native routes also enforce the wire format. /v1/responses requires a selection served on the OpenAI API, and /v1/messages one served on the Anthropic API. A selection qualifies when its Provider Assignment is backed by that provider type, or when the provider catalog declares the selected model's dialect as that format — Bedrock Mantle serves its openai-dialect models on /v1/responses and its anthropic-dialect models on /v1/messages, and Vertex AI serves its anthropic-dialect Claude models on /v1/messages. Provider Names may be custom labels; validation never infers anything from the label or the model name, and a model the catalog does not declare is rejected.

For example, this embedding request is evaluated against models.embeddings:

Code
{
  "model": "openai/text-embedding-3-small",
  "input": "Search text"
}

Adding fallbacks

Declare AI Gateway Fallback Model separately and place it immediately after this policy. Its fallback handles retryable errors and timeouts; quotaFallback handles usage-limit signals independently.

Write your own routing policy

Everything Model Filtering does is built on two public primitives, so a custom inbound policy can replace it entirely. The policy's job is to choose one allowed target and store it:

• AIGatewayModels.load(context) returns the cached provider catalog, including each model's capability, status, and per-token pricing.
• AIGatewayModelRouting.set(context, routing) validates the routing, resolves provider credentials internally, and stores the selection that the AI Gateway handler consumes.

Code
import {
  AIGatewayModelRouting,
  AIGatewayModels,
  type ZuploContext,
  type ZuploRequest,
} from "@zuplo/runtime";

export default async function selectModel(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const providers = await AIGatewayModels.load(context);
  const openAI = providers.find(
    (provider) => provider.providerName === "openai",
  );
  const model = openAI?.models.find(
    (candidate) =>
      candidate.capability === "completions" && candidate.status === "active",
  );
  if (!model) {
    throw new Error("No active OpenAI completions model is available");
  }

  await AIGatewayModelRouting.set(context, {
    completions: `openai/${model.model}`,
  });
  return request;
}

Attach the module as an ordinary inbound policy instead of Model Filtering:

Code
{
  "name": "my-model-filtering-inbound",
  "policyType": "custom-code-inbound",
  "handler": {
    "export": "default",
    "module": "$import(./modules/my-model-filtering)"
  }
}

AIGatewayModelRouting.get(context) returns the sanitized, normalized routing for the current request and never returns credentials. The AI Gateway handler consumes a custom selection even when Model Filtering is not attached. If neither kind of policy creates a selection, the handler derives one from the request's required providerName/model.

Policy order determines precedence:

1. Routing selected before Model Filtering remains authoritative because Model Filtering leaves an existing selection unchanged.
2. Model Filtering creates routing when no earlier policy selected it.
3. A custom policy placed after Model Filtering may deliberately replace that selection.
4. If no policy selects routing, the handler derives it from the request.

Prefer one policy as the primary selector so the route's intent is easy to understand.

Model discovery

Authenticated GET /v1/models and GET /v1/models/{providerName/model} use the same app URL as inference. Select the inference endpoint with the optional x-zuplo-models-endpoint request header:

Value	Eligible models	Response format
chat-completions	Active completion models supported by Chat Completions adapters, including existing translation	OpenAI
messages	Active completion models supported by native Messages passthrough	Anthropic
responses	Active completion models supported by the resolved Responses adapter	OpenAI
embeddings	Active embedding models supported by the resolved embedding adapter	OpenAI

An explicit header wins. Otherwise, the presence of anthropic-version selects messages; other requests default to chat-completions. Empty, unknown, and multiple selector values return an actionable 400. The selector affects discovery only. Embedding discovery is opt-in. Messages never translates to Chat Completions.

Discovery applies the first Model Filtering policy's parsed rules, matching inference's selection precedence. A capability omitted from that policy lists no models. Without Model Filtering, discovery lists the eligible catalog. Inactive models, unavailable adapters, and assignments without locally resolvable credentials are excluded. List and retrieve use identical eligibility rules; unknown, filtered-out, and wrong-endpoint IDs return the same 404.

IDs retain the exact model-name casing; only the provider assignment label is normalized. OpenAI entries contain id, object, created, and owned_by. owned_by means the provider assignment namespace, not the model's developer. Anthropic entries contain id, type, display_name, and created_at, with unknown required nullable metadata set to null. Unknown creation dates use the epoch; optional unknown metadata is omitted.

OpenAI lists return every eligible model and ignore pagination parameters. Anthropic lists default to 20 entries, accept limit from 1–1000, and accept either after_id or before_id from a previous page. Follow has_more and last_id for forward pagination. Invalid cursors return 400; restart without a cursor.

Discovery uses the existing 60-second catalog cache. An uncached lookup, including legacy fallback, has a two-second budget; failure returns a sanitized 503. A successfully loaded catalog with no eligible models returns an empty 200. Responses use Cache-Control: private, no-store and vary on both selector headers. Only GET is supported. POST, PUT, PATCH, DELETE, and HEAD return 405 with Allow: GET. OPTIONS is handled by the gateway’s existing CORS preflight handler before app routing.

Authentication and general abuse protections still apply. Discovery skips prompt inspection, semantic caching, and inference-budget blocking, and records one baseline request. Eligibility describes the catalog, adapters, and declared filtering rules; it does not guarantee upstream availability, provider acceptance of credentials, or acceptance by prompt-dependent custom policies. Existing Chat Completions translation does not promise full feature parity. Keep a qualified model configured when a client's picker does not consume discovery.

Read more about how policies work