AI Gateway: Spend Ceilings, Guardrails, and Source Control

This is the biggest AI Gateway update we have shipped so far. It adds spend attribution and limits for apps, teams, and gateways, alongside source-controlled projects, per-app policy chains, sensitive-data scanning, and new provider options.

• Usage limits at every level: Set daily or monthly limits on an app, a team, or the entire gateway
• Source-controlled projects: Review routes, policies, and custom code in your own repository
• Prompt and completion guardrails: Mask or block sensitive data, including values split across streamed responses
• More provider options: Add Bedrock Mantle and xAI, with updated Google and Mistral support

Usage limits for apps, teams, and gateways

Limits can count cost, tokens, or requests over a daily or monthly period. An app can have its own limit, its parent team can set another, and the gateway can enforce a final limit across every team underneath it.

The gateway checks each level on every request and stops at the first exceeded limit. App limits come from the Budgets and Costs policy. Team and gateway limits are enforced outside the app’s policy chain, so they still apply if an app owner reorders or disables local policies.

Each app chooses what happens next. The default is a 429 response that names the level, limit type, and period that blocked the request. You can also send over-budget traffic to a cheaper fallback model.

The Teams & Apps analytics view shows how those limits apply across the full hierarchy. Switch between cost, tokens, and requests, then compare daily or monthly usage at the gateway, team, or app level.

See the usage limits documentation for configuration details.

Source-controlled projects and per-app policy chains

Every AI Gateway project now deploys from a repository you own on GitHub, GitLab, Bitbucket, or Azure DevOps. Routes, policy declarations, and custom code can go through pull requests and rollbacks with the rest of your application.

config/policies.json declares the policies available in a project. In the Portal, each app selects which of those policies run and in what order. An app owner can reorder or disable an existing policy without a deployment, while a new policy or code change still goes through source control.

Teams can also define a starting policy chain for new apps. Entries can be locked when a policy must remain in place, and sub-teams inherit the template from their parent.

Read about the repository layout, policy chains, and policy templates.

Custom model routing in TypeScript

A custom policy is a TypeScript module in your repository. Once declared in config/policies.json, it can be added to an app’s chain like a built-in policy.

AIGatewayModels.load() returns the provider catalog with model capabilities, status, and per-token pricing, without exposing credentials. AIGatewayModelRouting.set() selects the provider and model for the current request. Together, they support price-aware routing, per-app model selection, and fallback orders defined in code.

The dynamic model routing cookbook includes a working example. See custom policies for the authoring model.

Sensitive-data detection for prompts and completions

The new Data Loss Prevention policy runs inside the gateway and scans both prompts and completions. It includes more than 80 rule types for PII, financial data, national identifiers, network addresses, and credentials from providers such as OpenAI, Anthropic, AWS, Stripe, and GitHub.

Each rule can mask, block, log, or ignore a match. Inbound and outbound actions are configured separately, and the scanner catches values split across chunks in a streamed response. The project template starts with PII and card numbers masked and secrets blocked in both directions.

The Data Loss Prevention policy documentation lists every rule and action.

Content guardrails with Akamai AI Firewall

The Akamai AI Firewall policy inspects prompts before they reach the provider and completions before they return to the caller. Streaming responses are supported.

The policy now fails closed by default. If the guardrail service is unavailable, the request returns a 502 instead of continuing without inspection. Set onError when availability should take priority for a particular app.

See the Akamai AI Firewall policy for setup and options.

Bedrock Mantle, xAI, and updated provider support

Bedrock Mantle and xAI are now available in the provider catalog. Bedrock Mantle supports OpenAI-compatible chat completions and embeddings, plus Claude on the native Messages endpoint. xAI provides Grok chat completions and embeddings under xai/<model>.

Google and Mistral now use their own OpenAI-compatible endpoints. Google requests can pass through tool calls, multimodal input, and JSON Schema structured output. Mistral requests support tools and JSON response formats.

You can also add the same provider more than once. Each configuration gets its own name, so separate OpenAI keys with different rate limits or cost centers can coexist in one gateway.

The provider documentation has the full list of supported models and endpoints.

Conversation-aware semantic caching

Semantic caching can now include several recent messages in its cache key. Set recentMessageCount to control how much of the latest conversation is considered, and use maxConversationLength to skip the cache when earlier context would make a match unsafe.

Entries are already separated by model, sampling parameters, and system prompt. Cache hits and misses now appear in the request log.

See the semantic cache policy for configuration details.

Observability in one place

Logs and analytics now share an Observability tab with cost, token, request, and cache breakdowns. Apps and teams appear in one tree, and each app includes a Playground for test requests plus copyable Chat Completions, Responses, and Messages endpoint URLs.

Tracing policies for Comet Opik and Galileo are also available in the project template.

Getting started

Create an AI Gateway in the Portal and send a request with one of the keyless demo models, or scaffold the project locally:

npx create-zuplo-api@latest my-ai-gateway --template ai-gateway

Follow the AI Gateway quickstart for the first request. The AI Gateway documentation covers providers, teams, apps, usage limits, and policies. For a closer look at how attribution and hierarchical limits work, read Attribute AI Spend to Teams and Apps.