Cookbook: Dynamic model routing

The Model Filtering policy covers static allow and block lists, but model selection can also be code. A custom chain policy can pick the model per request based on price, capability, the calling app, or anything else in the request. This recipe routes every request to the cheapest active completions model.

Two public primitives from @zuplo/runtime do the work:

• AIGatewayModels.load(context) returns the cached provider catalog, including each model's capability, status, and token rate card (tokenRateCard): the prices the gateway charges, which are the list prices with that provider's custom prices applied. Credentials are never returned.
• AIGatewayModelRouting.set(context, routing) validates the routing, resolves provider credentials internally, and stores the selection the AI Gateway handler uses. AIGatewayModelRouting.get(context) reads the current selection.

The policy

modules/cheapest-model.ts
import {
  AIGatewayModelInfo,
  AIGatewayModelRouting,
  AIGatewayModels,
  ZuploContext,
  ZuploRequest,
} from "@zuplo/runtime";

interface CheapestModelOptions {
  providers?: string[];
}

// The model's base input plus output price per token: the rates with no
// input-size tier or cache lifetime. Undefined when the model has no rate card
// or lacks either base rate.
function basePrice(model: AIGatewayModelInfo): number | undefined {
  const baseRate = (category: "input" | "output") =>
    model.tokenRateCard?.rates.find(
      (rate) =>
        rate.category === category &&
        rate.minInputTokens === undefined &&
        rate.maxInputTokens === undefined &&
        rate.cacheTtl === undefined,
    )?.costPerToken;
  const input = baseRate("input");
  const output = baseRate("output");
  return input === undefined || output === undefined
    ? undefined
    : input + output;
}

export default async function cheapestModel(
  request: ZuploRequest,
  context: ZuploContext,
  options: CheapestModelOptions,
): Promise<ZuploRequest | Response> {
  const allowed = options.providers ?? ["openai", "anthropic"];
  const candidates = (await AIGatewayModels.load(context))
    .filter(({ providerName }) => allowed.includes(providerName.toLowerCase()))
    .flatMap((provider) =>
      provider.models.map((model) => ({
        providerName: provider.providerName,
        model,
        price: basePrice(model),
      })),
    )
    .filter(
      (candidate): candidate is typeof candidate & { price: number } =>
        candidate.model.capability === "completions" &&
        candidate.model.status === "active" &&
        // Skip unpriced models instead of treating them as free.
        candidate.price !== undefined,
    )
    .sort((left, right) => left.price - right.price);

  const cheapest = candidates[0];
  if (!cheapest) {
    // Returning a Response short-circuits the chain and answers the request.
    return new Response("No eligible model", { status: 503 });
  }

  await AIGatewayModelRouting.set(context, {
    completions: `${cheapest.providerName}/${cheapest.model.model}`,
  });
  return request;
}

Declare and use it

Declare the module in config/policies.json and push:

config/policies.json (one entry in the policies array)
{
  "name": "cheapest-model",
  "policyType": "custom-code-inbound",
  "handler": {
    "export": "default",
    "module": "$import(./modules/cheapest-model)",
    "options": {
      "providers": ["openai", "anthropic"]
    }
  }
}

Then add cheapest-model to an app's chain on its Policies tab, in place of Model Filtering.

Routing precedence

Policy order determines which selection wins:

1. Routing selected before Model Filtering stays authoritative—Model Filtering leaves an existing selection unchanged.
2. Model Filtering creates routing when no earlier policy selected it.
3. A custom policy placed after Model Filtering may deliberately replace that selection.
4. If no policy selects routing, the handler derives it from the request's providerName/model.

Prefer one policy as the primary selector so the route's intent is easy to understand.

Next steps

• Custom Policies: the full custom-policy quickstart
• Cookbook: Custom fallback logic: enrich a selection instead of creating one
• Policy Chains: how chain order works