Question 1

How does Zuplo handle rate limiting for LLM API calls?

Accepted Answer

Zuplo supports request-based rate limiting with the rate-limit-inbound policy and usage quotas with the quota-inbound policy. Quotas can be keyed by API key, user, or any custom property and applied per minute, day, or month. Token-level accounting is implemented via custom TypeScript policies that read the token counts returned by your LLM provider.

Question 2

Can Zuplo act as an AI gateway in front of OpenAI, Anthropic, or other LLM providers?

Accepted Answer

Yes. Zuplo proxies requests to OpenAI, Anthropic, Gemini, and Mistral with authentication, rate limiting, cost controls, audit logging, prompt injection protection, and semantic caching as a transparent layer. Provider routing is configured declaratively — you can map requests across providers and define fallback behavior in your policy pipeline.

Question 3

How does Zuplo help control AI API costs?

Accepted Answer

Zuplo reduces AI costs through several built-in mechanisms: semantic caching returns cached responses for semantically similar queries, the quota-inbound policy caps consumption per consumer, and hierarchical team budgets set dollar spend limits. For cost-aware model routing (sending simpler prompts to cheaper models), teams use custom TypeScript policies — see our 'Control AI Costs' guide.

Question 4

How does Zuplo support token-based billing for AI products?

Accepted Answer

Zuplo can meter token consumption from LLM API responses and report usage to billing providers like Stripe. Custom TypeScript policies extract token counts from provider responses (e.g., OpenAI's usage object) and record them as billable units. This enables precise usage-based billing that reflects actual LLM cost drivers.

Question 5

How does semantic caching work in Zuplo for AI APIs?

Accepted Answer

Zuplo's semantic cache uses vector embeddings to identify requests that are semantically similar — not just identical — to previously cached queries. When a match is found above a configurable similarity threshold, the cached response is returned immediately without forwarding to the LLM. This reduces latency, cuts API costs, and improves throughput for AI-powered applications.

Question 6

How do we prevent models from acting beyond their intended scope?

Accepted Answer

This is an authorization problem, not a content one (OWASP LLM06). Zuplo addresses it at two layers. For LLM traffic, the AI Gateway constrains which providers, models, and routes a given API key can reach — keys can't call services outside their allowlist. For agent and tool traffic, MCP Gateway publishes virtual MCP servers that expose only the tools each team is approved to use, so the model literally cannot see — and therefore cannot invoke — tools outside its scope. Combined with auth translation at the gateway (the model never holds the privileged downstream credential) and per-tool RBAC, you get structural containment instead of trusting the model to behave.

Question 7

How can we monitor and limit costs when AI usage exceeds projections?

Accepted Answer

Token and dollar budgets at the API key level, with hierarchical caps that cascade across organizations, teams, sub-teams, and applications. Platform sets the ceiling, finance sets department caps, engineering divides budget across projects. Limits are enforced as hard 429s before the bill arrives — not soft alerts after — and per-team attribution is built in so you know exactly which workload is driving cost. Pair with semantic caching and per-consumer quotas to drop spend further.

Question 8

How can we prevent PII or SPDI from being shared in logs during troubleshooting?

Accepted Answer

Two distinct controls, deliberately separated. Inbound, the secret-masking-outbound policy redacts API keys, emails, and configurable patterns before requests reach the LLM provider, and Zuplo integrates with PII-detection and DLP providers (including Akamai's AI Firewall) for richer scanning. Outbound to your own observability stack, Zuplo scrubs sensitive fields at the logging-pipeline layer before shipping to your SIEM (Datadog, Splunk, S3, etc.). The right design depends on where your sensitive-data boundary sits — your platform team should map both the LLM provider and the log sink as separate trust zones.

Question 9

How do we monitor AI activity, especially for agents and AIOps?

Accepted Answer

Every LLM call and every MCP tool invocation is logged with caller identity, inputs, tool name, latency, tokens, cost, and outcome — and streamed to your SIEM. For AIOps and agent workloads specifically, the tool-call-level audit trail matters more than the LLM-call view: you want to see what the agent actually did, not just what it said. That tool-level audit is first-class in MCP Gateway. Trace data also flows to Galileo, Comet Opik, OpenTelemetry collectors, or your own pipeline for live latency and quality monitoring.

Question 10

How does Zuplo mitigate prompt injection attacks?

Accepted Answer

Honest framing first: prompt injection is the SQL injection of LLMs — you mitigate, you don't eliminate. Anyone selling a 'solved it' answer is overselling. Our approach is layered: (1) the prompt-injection-outbound policy, plus the optional Akamai AI Firewall integration, detects known injection patterns at the input; (2) MCP Gateway tool allowlists and scoped credentials limit blast radius if an injection succeeds — the worst the model can do is what it was already authorized to do; (3) outputs can be validated before they trigger downstream side effects; (4) every blocked attempt is logged for review.

Question 11

How do we prevent cross-team data leakage between AI workloads?

Accepted Answer

Virtual MCP servers per team, plus RBAC on the audit log itself. Finance gets the Stripe and QuickBooks tools; Engineering gets GitHub and Linear; Customer Success gets support tools — all from the same upstream sources, with team-specific configurations and isolated audit trails. One source of truth, separate views. The same hierarchy on the AI Gateway side scopes models, budgets, and request logs by team, so a finance application's logs aren't visible to an engineering API key.

Question 12

How do we stop developers from bypassing the gateway and calling providers like Bedrock directly?

Accepted Answer

Straight answer: no AI gateway alone fully solves this. It's a network-egress and IAM problem, and the gateway has to be paired with controls at those layers — block direct outbound calls to provider endpoints (Bedrock, Vertex, OpenAI), strip the corresponding permissions from developer identities so only the gateway's service principal has them, and vault provider keys so they never live in developer hands. Zuplo handles the key vaulting and is the enforcement point at the gateway layer; for the network and IAM side, our Akamai partnership is a real advantage if you're already in that ecosystem. We help platform teams design this end-to-end during onboarding.

Every LLM call, on the same gateway as your APIs.

Built for LLM traffic, governed like an API.

Multi-provider routing

Semantic caching

Prompt injection protection

Secret masking

Dollar budgets, per team

Observability & tracing

Want to see it in action?

Policies and budgets at every level of your org.

Route to the right model, every time.

Swap the base URL. Keep your SDK.

Not a standalone AI box — part of the unified gateway for APIs, AI, and MCP.

The questions every CISO asks about AI.

How do we prevent models from acting beyond their intended scope?

How do we cap costs when AI usage exceeds projections?

How do we keep PII and sensitive data out of logs and prompts?

How do we monitor AI activity, especially for agents and AIOps?

How do we mitigate "ignore all previous instructions" attacks?

How do we prevent one team's model from accessing another team's logs?

How do we stop developers from bypassing the gateway and calling Bedrock or OpenAI directly?

Frequently Asked Questions

Start with the AI Gateway today.