What Developers Want from an AI Gateway

Spend an afternoon in r/LocalLLaMA, r/LLMDevs, or any thread about AI gateways and the same wishlist surfaces over and over. Nobody’s asking for magic: they want one endpoint, predictable bills, and the freedom to swap models without a refactor.

We ranked the asks that come up most and marked each against what the Zuplo AI Gateway does today.

1. One endpoint, one key

The single most common ask: stop juggling a different SDK, base URL, and key for every provider. People want one endpoint any OpenAI-compatible client can point at, with one key they control.

This is the core of what Zuplo does. Keep your OpenAI SDK and swap the base URL to your gateway:

// client.ts
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://your-org.zuplo.app/v1", // your gateway, not api.openai.com
  apiKey: process.env.ZUPLO_AI_KEY, // your Zuplo key, never the provider's
});

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Summarize our Q1 sales data." }],
});

The key in your app is a Zuplo key. Your real OpenAI or Anthropic credentials are added once in the Zuplo portal and stay vaulted at the gateway, not scattered across services.

2. Routing, fallback, and load balancing

Right behind the unified endpoint, people want the gateway to decide where a request goes: route by model, fall back when a provider 500s, spread load across keys.

Zuplo handles provider routing declaratively. You map a route to OpenAI, Anthropic, Google, or Mistral, override the target per-request, per-tenant, or per-API-key, and define fallback with policies (the configurable request and response steps the gateway runs) rather than in application code. The app keeps calling the same endpoint while the gateway picks the upstream.

3. Cost and usage visibility

The recurring horror story is a surprise five-figure bill with no way to see which workload caused it. Developers want token counts, cost per call, and logs they can slice by team or feature.

Every request through the gateway is logged with caller identity, latency, tokens, and cost, with per-team attribution built in so you know which workload is spending. Trace data streams to observability tools like Galileo and Comet Opik, an OpenTelemetry collector, or your own pipeline, so the gateway feeds your existing stack instead of becoming a silo.

4. A broad model catalog

People want access to everything: the frontier labs, the cheap workhorses, and increasingly the open models like DeepSeek and Qwen that keep punching above their price.

Zuplo’s built-in providers are OpenAI, Anthropic, Google, and Mistral. Anything else with an OpenAI-compatible endpoint, DeepSeek, Qwen, or a model you host yourself, routes through the same way: point a route at its base URL and it behaves like any other upstream. The same budgets, logging, and rate limits apply, with no separate integration to wait for.

5. Low latency and boring reliability

A gateway sits in the hot path of every call, so the bar is that you shouldn’t feel it: streaming stays smooth and the proxy can’t be the thing that falls over.

Zuplo runs LLM traffic on the same edge runtime that serves its API gateway, and streaming is preserved so users see tokens as they arrive instead of waiting on a buffered response. Semantic caching (more below) also takes whole classes of repeated requests off the provider entirely.

6. Lower cost with no token markup

A frequent complaint about hosted gateways: they resell tokens at a margin, so you pay more than list price.

Zuplo’s model sidesteps that. You bring your own provider accounts and the gateway vaults those keys, so your billing relationship stays with OpenAI or Anthropic directly rather than running through a reseller. The gateway earns its keep by cutting spend, not by sitting on top of your token bill: caching and per-consumer quotas do the work.

7. Switch models without touching code

Closely tied to routing, but people ask for it constantly: try a new model, A/B two of them, or move off a provider after a price hike, all without a deploy.

Because the model is a config value and the endpoint never changes, you switch by editing config, not application code. That’s how the same setup drives Claude Code, Cursor, LangChain, and goose against whichever model you point them at, no deploy required.

8. Token-aware spend controls

Request-per-minute caps are the wrong meter for LLMs: one call can be 50 tokens or 50,000. The ask is budgets and quotas that understand tokens and dollars, with hard stops before the bill lands.

Zuplo does both. Hierarchical dollar budgets cascade from org to team to app, and token-aware rate limiting meters the real token counts from each upstream response. A hard stop is a real 429 the request gets back, not a soft alert that arrives after the money is gone. For the policy config, see our walkthrough on rate limiting LLM APIs by tokens, not requests.

9. Self-hosting and your own models

This is the one ask Zuplo answers differently, so here’s the honest version: Zuplo is a managed platform, not an open-source project you run on your own box. If a fully self-hosted, open-source gateway is a hard requirement, we’re probably not for you.

What Zuplo does cover is your own self-hosted models. A model served by vLLM (a common inference server) exposes an OpenAI-compatible endpoint, so the gateway fronts it like any other upstream, and enterprises that need single-tenant isolation can run on a dedicated deployment. Your weights stay where you put them; the gateway governs the traffic.

10. Multimodal, not just chat

Chat completions were table stakes a year ago. Now people want embeddings, images, audio, and rerank through the same gateway instead of standing up a second pipeline for each.

Embeddings are supported alongside chat and text completions. For other modalities, the gateway proxies the provider’s API rather than reshaping it, so image and audio requests can pass through to providers that accept them. The per-request controls still apply because they govern the call itself: the transparent proxy is the point, not a lowest-common-denominator wrapper.

11. Pay as you go, no lock-in

Developers are wary of annual contracts for something they’re still prototyping. The ask is to start free, pay for what you use, and leave without ripping out a proprietary SDK.

The Zuplo AI Gateway is free to start, and because you keep your own provider accounts and call a standard OpenAI-compatible endpoint, there’s no proprietary client to unwind if you leave. Your application code is the same code you’d write against the provider directly.

12. Prompt and response caching

Repeated and near-identical prompts are pure waste: you pay full price to regenerate an answer you already have. People want the gateway to catch those.

Zuplo’s semantic cache matches requests by vector similarity, not exact string match, so a reworded version of an earlier question can still hit the cache and skip the provider entirely. You set the similarity threshold that decides how close a match has to be before the cache answers instead of the model: tune it tight on routes where exact wording carries the meaning. It’s the same idea we unpack in what semantic caching actually is, and it cuts both latency and spend on the prompts that repeat.

13. Teams, RBAC, and SSO

Once more than one person uses the thing, the asks turn organizational: role-based access, per-team isolation, and SSO so access maps to your identity provider instead of a shared key in a Slack message.

Zuplo nests organizations, teams, sub-teams, and apps, each with their own budgets and policies that inherit down the tree. Access is role-based, identity comes from your IdP (Okta, Microsoft Entra, Auth0, or any OIDC provider), and one team’s request logs stay invisible to another team’s keys.

14. Pricing you can predict

Tied to half the others: people want to know what they’ll pay before the invoice, not after. Predictable, not just cheap.

This falls out of the spend controls. Hard budgets at every level mean the ceiling is something you set, not something you discover, and per-team attribution turns “where did the money go” into a dashboard rather than a forensic investigation. Combined with paying providers directly, the bill stops being a monthly surprise.

Where this leaves the list

Most of what developers ask for from an AI gateway is the same handful of things: one endpoint, control over routing and spend, visibility, and the freedom to swap models. Zuplo does the bulk of that out of the box, and where it doesn’t (a fully open-source, self-hosted build) we’d rather tell you than bury it.

If you’re weighing options, the AI gateway buyer’s guide walks the same criteria across the field. Or point your existing client at a Zuplo AI Gateway, route it to any provider you like, and check the list off yourself.