Copy page

Model Routing

Smart Router Policy

Classifies the last user prompt by calling a dedicated classifier AI Gateway app, stores the result on AIGatewaySmartRouter for later policies, and optionally routes completions by classified complexity.

Configuration

The configuration shows how to configure the policy in the 'policies.json' document.

Code
{
  "name": "my-ai-gateway-smart-router-inbound-policy",
  "policyType": "ai-gateway-smart-router-inbound",
  "handler": {
    "export": "AIGatewaySmartRouterInboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "classifierAppID": "$env(CLASSIFIER_APP_ID)",
      "classifierAppApiKey": "$env(CLASSIFIER_APP_API_KEY)",
      "classifierModel": "openai/gpt-4o-mini",
      "smartRoutingEnabled": true,
      "modelsByComplexity": {
        "low": "openai/gpt-4o-mini",
        "medium": "openai/gpt-4o",
        "high": "openai/gpt-5"
      }
    }
  }
}

Policy Configuration

• name <string> - The name of your policy instance. This is used as a reference in your routes.
• policyType <string> - The identifier of the policy. This is used by the Zuplo UI. Value should be ai-gateway-smart-router-inbound.
• handler.export <string> - The name of the exported type. Value should be AIGatewaySmartRouterInboundPolicy.
• handler.module <string> - The module containing the policy. Value should be $import(@zuplo/runtime).
• handler.options <object> - The options for this policy. See Policy Options below.

Policy Options

The options for this policy are specified below. All properties are optional unless specifically marked as required.

• classifierAppID (required) <string> - The AI Gateway application id used to run the classifier prompt and evaluate the user's request.
• classifierAppApiKey (required) <string> - API key sent as Authorization: Bearer when invoking the classifier app.
• classifierModel (required) <string> - The model (providerName/model) the classifier uses to evaluate the user's request. E.g. openai/gpt-4o-mini.
• smartRoutingEnabled <boolean> - When true, apply model routing based on the models set in modelsByComplexity when confidence score meets the minConfidenceForRouting threshold. When set to false, classification still runs but model routing is not applied, useful for debugging or testing the classifier prompt. Defaults to false.
• modelsByComplexity (required) <object> - Enables which model will be used based on the classifier results (low, medium and high). This configuration is applied only when smartRoutingEnabled is true and confidence score meets the minConfidenceForRouting threshold.
  • low (required) <string> - Model used for prompts classified as low complexity.
  • medium (required) <string> - Model used for prompts classified as medium complexity.
  • high (required) <string> - Model used for prompts classified as high complexity.
• intents <object[]> - Dictionary of intents used to classify the user message being evaluated. Omit to use the built-in dictionary (code, summarization, translation, qa, conversation, classification, creative_writing, agentic, document_qa, other).
  • id (required) <string> - Label used to classify the intent of the user message being evaluated.
  • description (required) <string> - Short description of the intent label.
• classifierPrompt <undefined> - Prompt used to analyze and classify the user message. Omit to use the built-in classifier prompt.
• minConfidenceForRouting <number> - Minimum confidence (0–1) required before applying model routing. Unknown intents are capped strictly below this threshold. Defaults to 0.5.
• classifierTimeoutMs <integer> - Timeout configured for the classifier task. When exceeded, the request is forwarded without classification to the original model. Defaults to 8000.
• maxPromptChars <integer> - Maximum characters of user message sent to the classifier. Longer messages get truncated. Defaults to 8000.

Using the Policy

Use this policy to classify the last user prompt on Chat Completions, Responses, and Anthropic Messages requests. Using the classification results, configure where to route the request based on its complexity.

When smartRoutingEnabled is true, it overwrites completions routing based on the configuration in modelsByComplexity.

Classification failure handling. If the message classification fails or times out, the request is forwarded to the original model.

Required options

• classifierAppID — AI Gateway application id whose chat/completions route runs the classifier.
• classifierAppApiKey — bearer token for that app. Use $env(CLASSIFIER_APP_API_KEY).
• classifierModel — providerName/model sent on the classifier request.
• modelsByComplexity — providerName/model for each of low, medium, and high. Used for routing when smartRoutingEnabled is true.

Omit intents and classifierPrompt to use the built-in dictionary (code, summarization, translation, qa, conversation, classification, creative_writing, agentic, document_qa, other) and the built-in classification prompt.

Example

Code
{
  "name": "ai-gateway-smart-router-inbound",
  "policyType": "ai-gateway-smart-router",
  "handler": {
    "export": "AIGatewaySmartRouterInboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "classifierAppID": "$env(CLASSIFIER_APP_ID)",
      "classifierAppApiKey": "$env(CLASSIFIER_APP_API_KEY)",
      "classifierModel": "openai/gpt-4o-mini",
      "smartRoutingEnabled": true,
      "modelsByComplexity": {
        "low": "openai/gpt-4o-mini",
        "medium": "openai/gpt-4o",
        "high": "openai/gpt-5"
      }
    }
  }
}

Policy order

Code
Model Filtering -> Smart Router -> Fallback Model -> AI Gateway handler

Smart Router always set()s routing when smart routing applies, even if filtering already selected a model. Filtering skips when routing is already set, so putting this policy first would also skip allow-list checks on the client's original model.

How classification drives routing

Complexity is classified independently of intent, as low, medium, or high. Smart Router looks up modelsByComplexity[complexity] and applies it only when all of the following hold:

• smartRoutingEnabled is true.
• The classified intent is a known one (not capped as an unknown intent).
• modelsByComplexity has a model configured for that complexity.
• profile.confidence >= minConfidenceForRouting (default 0.5).

When routing isn't applied, read smartRouting.reason from the result (see Using classification results in custom code) to see why: disabled, unknown-intent, no-model, low-confidence, internal-error, or applied.

Other advanced options

Option	Default	Purpose
minConfidenceForRouting	0.5	Raise it to route only on confident classifications; lower it to route more aggressively. Unknown intents are always capped just below this value, so they never qualify regardless of the setting.
classifierTimeoutMs	8000	How long to wait for the classifier before giving up and forwarding the request unclassified.
maxPromptChars	8000	Truncates the prompt sent to the classifier, to keep classifier cost and latency bounded on very long prompts.

Advanced configuration

Use these options to control what the classifier evaluates and how strongly its result influences routing.

Custom intents

intents replaces the built-in dictionary entirely — it's not additive. Provide a non-empty list of { id, description } pairs:

Code
{
  "intents": [
    {
      "id": "billing",
      "description": "Questions about invoices, payments, or subscription plans."
    },
    {
      "id": "support",
      "description": "Troubleshooting or how-to questions about the product."
    }
  ]
}

The id values become the enum the classifier model must return — the classifier calls a strict JSON-schema chat completion, so it can only return one of your configured ids. The description values are only shown to the classifier if your prompt includes {{intents}} (see below).

If the classifier returns an id outside this list, Smart Router keeps it as an "unknown intent" and caps its confidence just below minConfidenceForRouting, so it's never eligible for routing.

Custom classifier prompt

classifierPrompt replaces the built-in classifier prompt. It accepts either a string or an array of lines:

Code
{
  "classifierPrompt": [
    "You are an intent classifier for a billing support bot.",
    "Return JSON only that matches the schema.",
    "Intent (pick the single most specific match):",
    "{{intents}}"
  ]
}

Include the literal placeholder {{intents}} anywhere in the prompt to have it replaced with a - id: description line for each configured intent (or the built-in ones, if intents is also omitted). Pairing a custom intents list with the built-in classifierPrompt (by omitting classifierPrompt entirely) works out of the box, because the built-in prompt already contains {{intents}}.

Using classification results in custom code

AIGatewaySmartRouter.get(context) returns the result Smart Router stored on the request, or undefined if it didn't run (non-AI request, unreadable body, or a fail-open error):

Code
import { AIGatewaySmartRouter } from "@zuplo/runtime";

const result = AIGatewaySmartRouter.get(context);

result has this shape:

Code
interface AIGatewaySmartRouterResult {
  profile: {
    intent: string;
    complexity: "low" | "medium" | "high";
    confidence: number;
    reasons: string[];
  };
  usage: {
    promptTokens: number;
    completionTokens: number;
    totalTokens: number;
  };
  classifierModel: string;
  routing: { model?: string };
  durationMs: number;
  promptSource: "user" | "prior-user";
  promptLength: number;
  promptTruncated: boolean;
  unknownIntent: boolean;
  smartRouting: {
    enabled: boolean;
    applied: boolean;
    reason:
      | "applied"
      | "disabled"
      | "low-confidence"
      | "no-model"
      | "internal-error"
      | "unknown-intent";
    minConfidence: number;
  };
}

Read it from any policy placed after Smart Router in the chain. Common uses:

• Branch on intent or complexity — apply a stricter rate limit, a different DLP policy, or a longer timeout for high complexity or an agentic intent:

  Code
  const result = AIGatewaySmartRouter.get(context);
  if (result?.profile.complexity === "high") {
    // e.g. apply a stricter rate limit or route to a review queue
  }

• Layer business logic on top of modelsByComplexity — for example, cap the model for free-tier callers regardless of classified complexity. Read the plan from the caller's API key metadata, not from a raw request header (the caller controls headers and could set or omit them to bypass the cap). The built-in API Key Auth policy puts a key's metadata on request.user.data — set plan there when you create the key, and it lands on every request that key makes:

  Code
  import { AIGatewayModelRouting, AIGatewaySmartRouter } from "@zuplo/runtime";
  
  const result = AIGatewaySmartRouter.get(context);
  const isFreeTier = request.user?.data.plan === "free";
  if (result?.profile.complexity === "high" && isFreeTier) {
    await AIGatewayModelRouting.set(context, {
      completions: "openai/gpt-4o-mini",
    });
  }

• Observe why routing wasn't applied — log when smartRouting.reason is low-confidence or unknown-intent to tune minConfidenceForRouting or the intent taxonomy:

  Code
  const result = AIGatewaySmartRouter.get(context);
  if (result && !result.smartRouting.applied) {
    context.log.info(
      { reason: result.smartRouting.reason },
      "Smart routing skipped",
    );
  }

• Surface classification for debugging — add response headers in a non-production environment to see what the classifier returned. This runs in an outbound policy, so return a new Response carrying the headers — mutating a cloned Headers object alone has no effect on what the caller receives:

  Code
  const result = AIGatewaySmartRouter.get(context);
  if (!result) {
    return response;
  }
  
  const headers = new Headers(response.headers);
  headers.set("x-classified-intent", result.profile.intent);
  headers.set("x-classified-complexity", result.profile.complexity);
  
  return new Response(response.body, {
    headers,
    status: response.status,
    statusText: response.statusText,
  });

What content is evaluated

The policy reads the last user message in order to classify its intent and complexity. Embeddings, tool messages and other non-AI paths are skipped.

Fail-open behavior

The policy never 500s the user request for an internal classifier problem. Invalid options, classifier timeouts, empty classifier responses, and smart routing catalog errors are logged and the original request continues. Chat Completions, Responses, and Anthropic Messages are classified automatically. Embeddings and other non-AI paths are skipped.

Read more about how policies work