--- title: "AI Gateway — Control LLM Traffic" description: "Every LLM call, on the same gateway as your APIs. Get practical guidance, implementation details, and production tradeoffs for API and platform teams." canonicalUrl: "https://zuplo.com/ai-gateway" sourceUrl: "https://zuplo.com/ai-gateway" pageType: "product" generatedAt: "2026-09-23" --- # Every LLM call, on the same gateway as your APIs. > Right now, nothing is capping the bill or checking the prompts. Route your LLM > calls through the gateway that already runs your APIs — cap spend per team, > block abuse, and switch providers without touching code. **Start for Free** → https://portal.zuplo.com/signup **Talk to an AI Expert** → /schedule-call Trusted by companies including AccuWeather and Blockdaemon (customer logo strip, linking to case studies at `/customers/`). ## Features Built for LLM traffic, governed like an API. - **Multi-provider routing** — Route to OpenAI, Anthropic, Gemini, and Mistral through one endpoint. Swap providers in config, not code. - **Semantic caching** — Cache responses by vector similarity, not exact match. Cut latency and spend on repeated prompts. - **Prompt injection protection** — Malicious instructions get blocked before they ever reach your model. - **Secret masking** — Keys, emails, and PII are redacted before anyone — or anything — sees them. - **Dollar budgets, per team** — Per-team spend caps. Requests halt at the limit — no surprise bill. - **Observability & tracing** — Stream every request to Galileo, Comet Opik, or your own collector. Inspect latency, tokens, and cost per call. Want to see it in action? Book a Technical Call (AI gateway technical walkthrough). ## Team hierarchy Policies and budgets at every level of your org. Most AI gateways give you one global cap. Zuplo nests orgs, teams, and apps, each with its own budget — predictable spend, without anyone asking for a new key. - Dollar budgets at org, team, and app level - Daily and monthly limits — hard stops or warnings - Sub-team budgets cannot exceed the parent's ceiling - Rate limits and security policies inherit the same way **Start for Free** → https://portal.zuplo.com/signup ## Multi-provider routing Route to the right model, every time. One endpoint in front of every provider. Teams keep their SDK; platform teams decide which model actually serves each call. - Route to OpenAI, Anthropic, Gemini, or Mistral - Pin the model per route, tenant, or key - Keep the OpenAI SDK — just change the base URL - Works with Claude Code, Cursor, LangChain, Goose **Start for Free** → https://portal.zuplo.com/signup ## Drop-in integration Swap the base URL. Keep your SDK. Point any OpenAI-compatible client at your Zuplo gateway and policies take over. No new SDK to learn. ```typescript // client.ts // Keep your existing OpenAI SDK — just swap the base URL import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://your-org.zuplo.app/v1", apiKey: process.env.ZUPLO_AI_KEY, // your Zuplo key, not OpenAI's }); const response = await client.chat.completions.create({ model: "gpt-4o", messages: [{ role: "user", content: "Summarize our Q1 sales data." }], }); ``` **Start for Free** → https://portal.zuplo.com/signup ## Part of the platform Not a standalone AI box — part of the unified gateway for APIs, AI, and MCP. The AI Gateway runs on the same policy engine, auth, and GitOps pipeline as your REST and MCP traffic. One bill, one audit trail — when your AI product hits production, the gateway already knows how to run it. - Explore the platform → / - See every policy → https://zuplo.com/docs/policies ## Enterprise AI governance The questions every CISO asks about AI. Security, cost, and data-privacy controls a centralized AI governance program needs — and where Zuplo enforces each one. - **Unrestricted access** — _How do we prevent models from acting beyond their intended scope?_ Every key is scoped to approved models and tools. The model can't see anything else. → [MCP Gateway](/mcp-gateway) - **Cost control** — _How do we cap costs when AI usage exceeds projections?_ Dollar budgets per org, team, and app that hard-stop before the bill arrives. You always know which team is spending what. → [Usage limits](https://zuplo.com/docs/ai-gateway/usage-limits) - **Data privacy** — _How do we keep PII and sensitive data out of logs and prompts?_ Sensitive data is redacted before it reaches the model, and scrubbed again before logs leave. Two layers, on by default. → [AI Firewall](https://zuplo.com/docs/ai-gateway/policies/akamai-ai-firewall) - **Visibility & monitoring** — _How do we monitor AI activity, especially for agents and AIOps?_ Every call is logged with who, what, and how much — streamed to your SIEM. For agents, you see what they did, not just what they said. → [Audit logs](https://zuplo.com/docs/programmable-api/audit-log) - **Prompt injection** — _How do we mitigate "ignore all previous instructions" attacks?_ You mitigate, not eliminate: detect attacks at the door, and cap the damage with tool allowlists if one gets through. → [AI Firewall](https://zuplo.com/docs/ai-gateway/policies/akamai-ai-firewall) - **Cross-team data leakage** — _How do we prevent one team's model from accessing another team's logs?_ Each team gets its own tools and its own logs. Finance sees Stripe, Engineering sees GitHub — never each other. → [MCP Gateway](/mcp-gateway) - **Unauthorized access** — _How do we stop developers from bypassing the gateway and calling Bedrock or OpenAI directly?_ Honest answer: no gateway alone does. Zuplo is the enforcement point and key vault; lock down network egress and IAM, and our Akamai partnership covers the rest. → [Dedicated deployment](https://zuplo.com/docs/dedicated/akamai/ai-powered-applications) ## FAQ **How does Zuplo handle rate limiting for LLM API calls?** Zuplo supports request-based rate limiting with the rate-limit-inbound policy and usage quotas with the quota-inbound policy. Quotas can be keyed by API key, user, or any custom property and applied per minute, day, or month. Token-level accounting is implemented via custom TypeScript policies that read the token counts returned by your LLM provider. **Can Zuplo act as an AI gateway in front of OpenAI, Anthropic, or other LLM providers?** Yes. Zuplo proxies requests to OpenAI, Anthropic, Gemini, and Mistral with authentication, rate limiting, cost controls, audit logging, prompt injection protection, and semantic caching as a transparent layer. Provider routing is configured declaratively — you can map requests across providers and define fallback behavior in your policy pipeline. **How does Zuplo help control AI API costs?** Zuplo reduces AI costs through several built-in mechanisms: semantic caching returns cached responses for semantically similar queries, the quota-inbound policy caps consumption per consumer, and hierarchical team budgets set dollar spend limits. For cost-aware model routing (sending simpler prompts to cheaper models), teams use custom TypeScript policies — see our "Control AI Costs" guide. **How does Zuplo support token-based billing for AI products?** Zuplo can meter token consumption from LLM API responses and report usage to billing providers like Stripe. Custom TypeScript policies extract token counts from provider responses (e.g., OpenAI's usage object) and record them as billable units. This enables precise usage-based billing that reflects actual LLM cost drivers. **How does semantic caching work in Zuplo for AI APIs?** Zuplo's semantic cache uses vector embeddings to identify requests that are semantically similar — not just identical — to previously cached queries. When a match is found above a configurable similarity threshold, the cached response is returned immediately without forwarding to the LLM. This reduces latency, cuts API costs, and improves throughput for AI-powered applications. ## Next steps - Start with the AI Gateway for free: https://portal.zuplo.com/signup - Book a technical walkthrough call - Compare AI Gateways: /learning-center/best-ai-gateway-buyers-guide - Same signup as the API Gateway — your account owns both.