Cookbook: Dynamic model routing

The Model Filtering policy covers static allow and block lists, but model selection can also be code. A custom chain policy can pick the model per request based on price, capability, the calling app, or anything else in the request. This recipe routes every request to the cheapest active completions model.

Two public primitives from @zuplo/runtime do the work:

• AIGatewayModels.load(context) returns the cached provider catalog, including each model's capability, status, and per-token pricing. Credentials are never returned.
• AIGatewayModelRouting.set(context, routing) validates the routing, resolves provider credentials internally, and stores the selection the AI Gateway handler uses. AIGatewayModelRouting.get(context) reads the current selection.

The policy

modules/cheapest-model.ts
import {
  AIGatewayModelRouting,
  AIGatewayModels,
  ZuploContext,
  ZuploRequest,
} from "@zuplo/runtime";

interface CheapestModelOptions {
  providers?: string[];
}

export default async function cheapestModel(
  request: ZuploRequest,
  context: ZuploContext,
  options: CheapestModelOptions,
): Promise<ZuploRequest | Response> {
  const allowed = options.providers ?? ["openai", "anthropic"];
  const candidates = (await AIGatewayModels.load(context))
    .filter(({ providerName }) => allowed.includes(providerName.toLowerCase()))
    .flatMap((provider) =>
      provider.models.map((model) => ({
        providerName: provider.providerName,
        model,
      })),
    )
    .filter(
      ({ model }) =>
        model.capability === "completions" && model.status === "active",
    )
    .sort(
      (left, right) =>
        left.model.inputCostPerToken +
        left.model.outputCostPerToken -
        (right.model.inputCostPerToken + right.model.outputCostPerToken),
    );

  const cheapest = candidates[0];
  if (!cheapest) {
    // Returning a Response short-circuits the chain and answers the request.
    return new Response("No eligible model", { status: 503 });
  }

  await AIGatewayModelRouting.set(context, {
    completions: `${cheapest.providerName}/${cheapest.model.model}`,
  });
  return request;
}

Declare and use it

Declare the module in config/policies.json and push:

config/policies.json (one entry in the policies array)
{
  "name": "cheapest-model",
  "policyType": "custom-code-inbound",
  "handler": {
    "export": "default",
    "module": "$import(./modules/cheapest-model)",
    "options": {
      "providers": ["openai", "anthropic"]
    }
  }
}

Then add cheapest-model to an app's chain on its Policies tab, in place of Model Filtering.

Routing precedence

Policy order determines which selection wins:

1. Routing selected before Model Filtering stays authoritative—Model Filtering leaves an existing selection unchanged.
2. Model Filtering creates routing when no earlier policy selected it.
3. A custom policy placed after Model Filtering may deliberately replace that selection.
4. If no policy selects routing, the handler derives it from the request's providerName/model.

Prefer one policy as the primary selector so the route's intent is easy to understand.

Next steps

• Custom Policies: the full custom-policy quickstart
• Cookbook: Custom fallback logic: enrich a selection instead of creating one
• Policy Chains: how chain order works