---
title: "AI Gateway — Control LLM Traffic"
description:
  "Every LLM call, on the same gateway as your APIs. Route between providers,
  cache semantically, block prompt injections, and cap dollar spend per team —
  with one programmable policy engine."
canonicalUrl: "https://zuplo.com/ai-gateway"
sourceUrl: "https://zuplo.com/ai-gateway"
pageType: "product"
generatedAt: "2026-09-01"
---

# Every LLM call, on the same gateway as your APIs.

> Right now, nothing is capping the bill or checking the prompts. Route your LLM
> calls through the gateway that already runs your APIs — cap spend per team,
> block abuse, and switch providers without touching code.

Trusted by companies including AccuWeather, Blockdaemon, Lake Michigan Credit
Union, Finsolutia, VNDR, Mews, Yext, and Zumiez.

## Built for LLM traffic, governed like an API

- **Multi-provider routing** — Route to OpenAI, Anthropic, Gemini, and Mistral
  through one endpoint. Swap providers in config, not code.
- **Semantic caching** — Cache responses by vector similarity, not exact match.
  Cut latency and spend on repeated prompts.
- **Prompt injection protection** — Malicious instructions get blocked before
  they ever reach your model.
- **Secret masking** — Keys, emails, and PII are redacted before anyone — or
  anything — sees them.
- **Dollar budgets, per team** — Per-team spend caps. Requests halt at the limit
  — no surprise bill.
- **Observability & tracing** — Stream every request to Galileo, Comet Opik, or
  your own collector. Inspect latency, tokens, and cost per call.

## Policies and budgets at every level of your org

Most AI gateways give you one global cap. Zuplo nests orgs, teams, and apps,
each with its own budget — predictable spend, without anyone asking for a new
key.

- Dollar budgets at org, team, and app level
- Daily and monthly limits — hard stops or warnings
- Sub-team budgets cannot exceed the parent's ceiling
- Rate limits and security policies inherit the same way

The hierarchy visual on the page shows an example: Acme Inc (organization, hard
cap $4,393 / $10,000) breaking down into Engineering ($2,210 / $4,000, itself
split into Chatbot at $670 / $800 and Copilot at $1,540 / $3,200), Sales
($1,040 / $3,000), and Marketing ($1,143 / $3,000).

## Route to the right model, every time

One endpoint in front of every provider. Teams keep their SDK; platform teams
decide which model actually serves each call.

- Route to OpenAI, Anthropic, Gemini, or Mistral
- Pin the model per route, tenant, or key
- Keep the OpenAI SDK — just change the base URL
- Works with Claude Code, Cursor, LangChain, Goose

## Swap the base URL. Keep your SDK.

Point any OpenAI-compatible client at your Zuplo gateway and policies take over.
No new SDK to learn.

```typescript
// Keep your existing OpenAI SDK — just swap the base URL
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://your-org.zuplo.app/v1",
  apiKey: process.env.ZUPLO_AI_KEY, // your Zuplo key, not OpenAI's
});

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Summarize our Q1 sales data." }],
});
```

## Not a standalone AI box — part of the unified gateway for APIs, AI, and MCP

The AI Gateway runs on the same policy engine, auth, and GitOps pipeline as your
REST and MCP traffic. One bill, one audit trail — when your AI product hits
production, the gateway already knows how to run it.

- [Explore the platform](/)
- [See every policy](https://zuplo.com/docs/policies)

## The questions every CISO asks about AI

Security, cost, and data-privacy controls a centralized AI governance program
needs — and where Zuplo enforces each one.

**Unrestricted access** — How do we prevent models from acting beyond their
intended scope? Every key is scoped to approved models and tools. The model
can't see anything else. [MCP Gateway →](/mcp-gateway)

**Cost control** — How do we cap costs when AI usage exceeds projections? Dollar
budgets per org, team, and app that hard-stop before the bill arrives. You
always know which team is spending what.
[Usage limits →](https://zuplo.com/docs/ai-gateway/usage-limits)

**Data privacy** — How do we keep PII and sensitive data out of logs and
prompts? Sensitive data is redacted before it reaches the model, and scrubbed
again before logs leave. Two layers, on by default.
[AI Firewall →](https://zuplo.com/docs/ai-gateway/policies/akamai-ai-firewall)

**Visibility & monitoring** — How do we monitor AI activity, especially for
agents and AIOps? Every call is logged with who, what, and how much — streamed
to your SIEM. For agents, you see what they did, not just what they said.
[Audit logs →](https://zuplo.com/docs/programmable-api/audit-log)

**Prompt injection** — How do we mitigate "ignore all previous instructions"
attacks? You mitigate, not eliminate: detect attacks at the door, and cap the
damage with tool allowlists if one gets through.
[AI Firewall →](https://zuplo.com/docs/ai-gateway/policies/akamai-ai-firewall)

**Cross-team data leakage** — How do we prevent one team's model from accessing
another team's logs? Each team gets its own tools and its own logs. Finance sees
Stripe, Engineering sees GitHub — never each other.
[MCP Gateway →](/mcp-gateway)

**Unauthorized access** — How do we stop developers from bypassing the gateway
and calling Bedrock or OpenAI directly? Honest answer: no gateway alone does.
Zuplo is the enforcement point and key vault; lock down network egress and IAM,
and our Akamai partnership covers the rest.
[Dedicated deployment →](https://zuplo.com/docs/dedicated/akamai/ai-powered-applications)

## FAQ

**How does Zuplo handle rate limiting for LLM API calls?** Zuplo supports
request-based rate limiting with the rate-limit-inbound policy and usage quotas
with the quota-inbound policy. Quotas can be keyed by API key, user, or any
custom property and applied per minute, day, or month. Token-level accounting is
implemented via custom TypeScript policies that read the token counts returned
by your LLM provider.

**Can Zuplo act as an AI gateway in front of OpenAI, Anthropic, or other LLM
providers?** Yes. Zuplo proxies requests to OpenAI, Anthropic, Gemini, and
Mistral with authentication, rate limiting, cost controls, audit logging, prompt
injection protection, and semantic caching as a transparent layer. Provider
routing is configured declaratively — you can map requests across providers and
define fallback behavior in your policy pipeline.

**How does Zuplo help control AI API costs?** Zuplo reduces AI costs through
several built-in mechanisms: semantic caching returns cached responses for
semantically similar queries, the quota-inbound policy caps consumption per
consumer, and hierarchical team budgets set dollar spend limits. For cost-aware
model routing (sending simpler prompts to cheaper models), teams use custom
TypeScript policies — see our "Control AI Costs" guide.

**How does Zuplo support token-based billing for AI products?** Zuplo can meter
token consumption from LLM API responses and report usage to billing providers
like Stripe. Custom TypeScript policies extract token counts from provider
responses (e.g., OpenAI's usage object) and record them as billable units. This
enables precise usage-based billing that reflects actual LLM cost drivers.

**How does semantic caching work in Zuplo for AI APIs?** Zuplo's semantic cache
uses vector embeddings to identify requests that are semantically similar — not
just identical — to previously cached queries. When a match is found above a
configurable similarity threshold, the cached response is returned immediately
without forwarding to the LLM. This reduces latency, cuts API costs, and
improves throughput for AI-powered applications.

## Next steps

- Start with the AI Gateway today — free forever for developers, same signup as
  the API Gateway ([Start for Free](https://portal.zuplo.com/signup))
- [Compare AI Gateways](/learning-center/best-ai-gateway-buyers-guide.md)
- [Talk to an AI expert](https://portal.zuplo.com/signup) for a technical
  walkthrough