---
title: "What Developers Want from an AI Gateway"
description: "We read the AI dev corners of Reddit and ranked the 14 things people actually want from an AI gateway, with where the Zuplo AI Gateway delivers (and where it doesn't)."
canonicalUrl: "https://zuplo.com/blog/2026/07/08/what-developers-want-from-an-ai-gateway"
pageType: "blog"
date: "2026-07-08"
authors: "martyn"
tags: "ai-gateway, llm"
image: "https://zuplo.com/og?text=What%20Developers%20Want%20from%20an%20AI%20Gateway"
---
Spend an afternoon in r/LocalLLaMA, r/LLMDevs, or any thread about AI gateways
and the same wishlist surfaces over and over. Nobody's asking for magic: they
want one endpoint, predictable bills, and the freedom to swap models without a
refactor.
We ranked the asks that come up most and marked each against what the
[Zuplo AI Gateway](https://zuplo.com/ai-gateway) does today.
## 1. One endpoint, one key
The single most common ask: stop juggling a different SDK, base URL, and key for
every provider. People want one endpoint any OpenAI-compatible client can point
at, with one key they control.
This is the core of what Zuplo does. Keep your OpenAI SDK and swap the base URL
to your gateway:
```ts
// client.ts
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://your-org.zuplo.app/v1", // your gateway, not api.openai.com
apiKey: process.env.ZUPLO_AI_KEY, // your Zuplo key, never the provider's
});
const response = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "Summarize our Q1 sales data." }],
});
```
The key in your app is a Zuplo key. Your real OpenAI or Anthropic credentials
are added once in the Zuplo portal and stay vaulted at the gateway, not
scattered across services.
## 2. Routing, fallback, and load balancing
Right behind the unified endpoint, people want the gateway to decide where a
request goes: route by model, fall back when a provider 500s, spread load across
keys.
Zuplo handles [provider routing](https://zuplo.com/docs/ai-gateway/providers)
declaratively. You map a route to OpenAI, Anthropic, Google, or Mistral,
override the target per-request, per-tenant, or per-API-key, and define
[fallback](https://zuplo.com/docs/ai-gateway/fallback) with policies (the
configurable request and response steps the gateway runs) rather than in
application code. The app keeps calling the same endpoint while the gateway
picks the upstream.
## 3. Cost and usage visibility
The recurring horror story is a surprise five-figure bill with no way to see
which workload caused it. Developers want token counts, cost per call, and logs
they can slice by team or feature.
Every request through the gateway is logged with caller identity, latency,
tokens, and cost, with per-team attribution built in so you know which workload
is spending. Trace data streams to observability tools like
[Galileo](https://zuplo.com/docs/ai-gateway/policies/galileo-tracing) and
[Comet Opik](https://zuplo.com/docs/ai-gateway/policies/comet-opik-tracing), an
OpenTelemetry collector, or your own pipeline, so the gateway feeds your
existing stack instead of becoming a silo.
## 4. A broad model catalog
People want access to everything: the frontier labs, the cheap workhorses, and
increasingly the open models like DeepSeek and Qwen that keep punching above
their price.
Zuplo's [built-in providers](https://zuplo.com/docs/ai-gateway/providers) are
OpenAI, Anthropic, Google, and Mistral. Anything else with an OpenAI-compatible
endpoint, DeepSeek, Qwen, or
[a model you host yourself](https://zuplo.com/docs/ai-gateway/custom-providers),
routes through the same way: point a route at its base URL and it behaves like
any other upstream. The same budgets, logging, and rate limits apply, with no
separate integration to wait for.
## 5. Low latency and boring reliability
A gateway sits in the hot path of every call, so the bar is that you shouldn't
feel it: streaming stays smooth and the proxy can't be the thing that falls
over.
Zuplo runs LLM traffic on the same edge runtime that serves its API gateway, and
streaming is preserved so users see tokens as they arrive instead of waiting on
a buffered response. Semantic caching (more below) also takes whole classes of
repeated requests off the provider entirely.
## 6. Lower cost with no token markup
A frequent complaint about hosted gateways: they resell tokens at a margin, so
you pay more than list price.
Zuplo's model sidesteps that. You bring your own provider accounts and the
gateway vaults those keys, so your billing relationship stays with OpenAI or
Anthropic directly rather than running through a reseller. The gateway earns its
keep by cutting spend, not by sitting on top of your token bill: caching and
per-consumer quotas do the work.
## 7. Switch models without touching code
Closely tied to routing, but people ask for it constantly: try a new model, A/B
two of them, or move off a provider after a price hike, all without a deploy.
Because the model is a config value and the endpoint never changes, you switch
by editing config, not application code. That's how the same setup drives
[Claude Code, Cursor, LangChain, and goose](https://zuplo.com/blog/ai-gateway-with-claude-code)
against whichever model you point them at, no deploy required.
## 8. Token-aware spend controls
Request-per-minute caps are the wrong meter for LLMs: one call can be 50 tokens
or 50,000. The ask is budgets and quotas that understand tokens and dollars,
with hard stops before the bill lands.
Zuplo does both. Hierarchical
[dollar budgets](https://zuplo.com/docs/ai-gateway/usage-limits) cascade from
org to team to app, and token-aware rate limiting meters the real token counts
from each upstream response. A hard stop is a real `429` the request gets back,
not a soft alert that arrives after the money is gone. For the policy config,
see our walkthrough on
[rate limiting LLM APIs by tokens, not requests](https://zuplo.com/blog/rate-limit-llm-apis-by-tokens-not-requests).
## 9. Self-hosting and your own models
This is the one ask Zuplo answers differently, so here's the honest version:
Zuplo is a managed platform, not an open-source project you run on your own box.
If a fully self-hosted, open-source gateway is a hard requirement, we're
probably not for you.
What Zuplo does cover is
[your own self-hosted models](https://zuplo.com/docs/ai-gateway/custom-providers).
A model served by vLLM (a common inference server) exposes an OpenAI-compatible
endpoint, so the gateway fronts it like any other upstream, and enterprises that
need single-tenant isolation can run on a
[dedicated deployment](https://zuplo.com/docs/dedicated/akamai/ai-powered-applications).
Your weights stay where you put them; the gateway governs the traffic.
## 10. Multimodal, not just chat
Chat completions were table stakes a year ago. Now people want embeddings,
images, audio, and rerank through the same gateway instead of standing up a
second pipeline for each.
Embeddings are supported alongside chat and text completions. For other
modalities, the gateway proxies the provider's API rather than reshaping it, so
image and audio requests can pass through to providers that accept them. The
per-request controls still apply because they govern the call itself: the
transparent proxy is the point, not a lowest-common-denominator wrapper.
## 11. Pay as you go, no lock-in
Developers are wary of annual contracts for something they're still prototyping.
The ask is to start free, pay for what you use, and leave without ripping out a
proprietary SDK.
The Zuplo AI Gateway is free to start, and because you keep your own provider
accounts and call a standard OpenAI-compatible endpoint, there's no proprietary
client to unwind if you leave. Your application code is the same code you'd
write against the provider directly.
## 12. Prompt and response caching
Repeated and near-identical prompts are pure waste: you pay full price to
regenerate an answer you already have. People want the gateway to catch those.
Zuplo's semantic cache matches requests by vector similarity, not exact string
match, so a reworded version of an earlier question can still hit the cache and
skip the provider entirely. You set the similarity threshold that decides how
close a match has to be before the cache answers instead of the model: tune it
tight on routes where exact wording carries the meaning. It's the same idea we
unpack in
[what semantic caching actually is](https://zuplo.com/blog/what-is-semantic-caching),
and it cuts both latency and spend on the prompts that repeat.
## 13. Teams, RBAC, and SSO
Once more than one person uses the thing, the asks turn organizational:
role-based access, per-team isolation, and SSO so access maps to your identity
provider instead of a shared key in a Slack message.
Zuplo nests
[organizations, teams, sub-teams, and apps](https://zuplo.com/docs/ai-gateway/teams),
each with their own budgets and policies that inherit down the tree. Access is
role-based, identity comes from your IdP (Okta, Microsoft Entra, Auth0, or any
OIDC provider), and one team's request logs stay invisible to another team's
keys.
## 14. Pricing you can predict
Tied to half the others: people want to know what they'll pay before the
invoice, not after. Predictable, not just cheap.
This falls out of the spend controls. Hard budgets at every level mean the
ceiling is something you set, not something you discover, and per-team
attribution turns "where did the money go" into a dashboard rather than a
forensic investigation. Combined with paying providers directly, the bill stops
being a monthly surprise.
## Where this leaves the list
Most of what developers ask for from an AI gateway is the same handful of
things: one endpoint, control over routing and spend, visibility, and the
freedom to swap models. Zuplo does the bulk of that out of the box, and where it
doesn't (a fully open-source, self-hosted build) we'd rather tell you than bury
it.
If you're weighing options, the
[AI gateway buyer's guide](https://zuplo.com/learning-center/best-ai-gateway-buyers-guide)
walks the same criteria across the field. Or point your existing client at a
[Zuplo AI Gateway](https://zuplo.com/ai-gateway), route it to any provider you
like, and check the list off yourself.