Your provider invoice can tell you how much you spent on OpenAI, Anthropic, or Bedrock. It cannot tell you which team spent it, which app inside that team ran up the bill, or whether that app should still be allowed to make another call.
The latest AI Gateway release fills in that missing ownership layer. Usage and cost now roll up from apps to teams to the gateway, with limits enforced at each level. We also added source-controlled gateway projects, per-app policy chains, sensitive-data scanning, more provider options, and better semantic caching for conversations.
It is the biggest update we have made to AI Gateway so far. Here is what changed and why it matters.
- Several teams or apps share the same AI providers
- You need spend limits above the level of a single app
- Prompts or completions may contain PII or credentials
- You use Bedrock or Anthropic and want to keep their native API shapes
See AI spend by app and team
Each app in a gateway gets its own API key. When a request arrives, the API Key Authentication policy resolves that key to the app and carries the identity through the request. Cost, token counts, and usage are recorded against the app instead of disappearing into one organization-wide total.
Apps can match the units you already work with: a service, an environment, a product, or a squad. Teams group those apps under the people who own the budget.
The calling app is also available to custom policies through request.user.sub,
so you can route models or apply rules based on who is calling. New top-level
teams include API key authentication as a locked policy template entry, which
keeps that identity layer in place as teams add apps.
Set spend limits at the gateway, team, or app level
You can now set daily or monthly limits for cost, tokens, or requests on an app, its team, and the gateway as a whole. Warning thresholds let owners know a budget is getting close before traffic is affected.
The gateway checks every level on each request, starting with the app and walking up to the root. It stops at the first limit that has been exceeded.
App limits come from the Budgets and Costs policy in the app’s chain. Team and gateway limits are enforced outside that chain. This matters because an app owner can reorder or disable local policies, but cannot accidentally remove a limit set by the platform team.
When a limit is reached, the app can return a 429 or fall back to a cheaper
model. The error response names the level, limit type, and period that blocked
the request. With a quota fallback, the app keeps working at a lower cost
instead of going dark.
AI Gateway projects now live in your repository
Every AI Gateway project now deploys from a repository you own. Routes, custom code, and policy declarations can go through pull requests and rollbacks with the rest of your application.
The project declares its available policies in config/policies.json. Each app
then chooses which policies run and in what order from the Portal. Existing
policies can be reordered or disabled without a deployment, while adding a new
policy or changing its code still goes through source control.
Teams can define a starting chain for every new app and lock the entries that must remain in place. Sub-teams inherit that template from their parent. This gives platform teams a baseline without forcing every app to have an identical chain.
You can also scaffold an AI Gateway project locally:
Custom policies make model routing programmable
When the built-in policies do not match your routing rules, you can write a policy in TypeScript and add it to the same per-app chain.
Two new runtime APIs handle model selection. AIGatewayModels.load() returns
the available models with their capabilities, status, and per-token pricing,
without exposing provider credentials. AIGatewayModelRouting.set() chooses the
provider and model for the current request.
That is enough to route an app to the cheapest active model, prefer a provider in a particular region, or build a fallback order around your own cost and reliability requirements. The dynamic model routing cookbook has a complete example.
Scan prompts and completions for sensitive data
Budget controls tell you who is spending money. They do not stop someone from pasting an API key into a prompt or a model from returning sensitive data.
This release adds two inspection options that cover different risks:
| Akamai AI Firewall | Data Loss Prevention | |
|---|---|---|
| Detection | A guardrail service inspects prompts and completions, including streams | Pattern rules run inside the gateway without a third-party scanner |
| Coverage | Content-safety guardrails | 80+ rule types for PII, financial data, identifiers, addresses, and provider credentials |
| Per-rule action | Pass or block | Mask, block, log, or off |
| Direction | Both directions in one policy | Separate inbound and outbound rule sets |
| Failure mode | Returns a 502 when the guardrail service is unavailable |
Catches secrets split across streamed chunks |
The project template starts with PII and card numbers masked and secrets blocked in both directions. The credential rules recognize key formats from OpenAI, Anthropic, AWS Bedrock, Stripe, GitHub, and other common providers.
Data Loss Prevention policy
See the full rule catalog and choose how inbound and outbound matches are handled.
Use Bedrock Mantle, Grok, and more native provider endpoints
Teams that are already committed to a provider do not always want to rewrite their clients around an OpenAI-shaped API. We heard that especially often from Bedrock and Anthropic users.
| Provider | What changed |
|---|---|
| Bedrock Mantle | OpenAI-compatible chat completions and embeddings, plus Claude on the native Messages endpoint, using your Bedrock long-term key |
| xAI | Grok chat completions and embeddings under xai/<model> |
| Gemini’s own OpenAI-compatible endpoint, including tool calling, multimodal input, and JSON Schema structured output | |
| Mistral | Mistral’s own OpenAI-compatible chat completions endpoint, including tools and JSON response formats |
| Zuplo demo models | Three keyless persona models, so you can send a first request before configuring a provider key |
You can now add the same provider more than once as well. Each configuration gets its own name, so two OpenAI keys with different rate limits or cost centers can coexist in one gateway.
Semantic caching now accounts for recent conversation turns
A cache key based only on the latest message is not very useful for a conversation. The same follow-up question can mean something completely different depending on the previous turns.
Set recentMessageCount to include the latest part of the conversation in the
cache key. maxConversationLength tells the cache to sit out when the history
gets long enough that earlier context should affect the answer.
Entries remain separated by model, sampling parameters, and system prompt. An exact repeat can still skip the embedding call and vector lookup entirely.
User-level identity comes next
This release attributes usage to an app. The next step is identifying the person behind the request, so spend and policy can follow a user from your identity provider across apps.
We are testing user-level identity with a small group of teams before opening it more broadly.
User-level identity in the AI Gateway
Attribute spend and apply policy per user from your identity provider, not just per app.
Try the release
If several services share provider keys today, start with one app key per service and a gateway-wide monthly cost limit. That immediately gives you spend by workload and a hard ceiling on the total.
The keyless demo models also make it possible to send a first request before setting up a provider. Since adding those models and shipping a scaffold with the policy menu already declared, the share of new gateways sending a live request has increased from roughly 6% to 45%. Median time from signup to first request is about 17 minutes.
Follow the AI Gateway quickstart to create a gateway and send that first request. The AI Gateway documentation covers providers, teams, apps, usage limits, and policy chains. The release notes have the complete list of changes.