How to Use the OpenAI SDK with Rate Limits and Cost Caps Behind a Gateway

Your platform team wants spend caps, token limits, and an audit trail on every LLM call. Your product teams want to keep shipping with the OpenAI SDK they already know. Those two goals sound like they’re in tension, and in a lot of organizations they get resolved badly — either with a hand-rolled internal wrapper library nobody maintains, or with no controls at all until the first surprise invoice lands.

There’s a much cheaper resolution. Because the OpenAI SDK exposes a baseURL option, you can put a gateway in the request path without touching a single line of application logic. Teams keep the same client, the same messages array, and the same parameters. The gateway sees every request and enforces token limits, USD spend caps, and security policies before anything reaches a provider.

This guide walks through the change end to end: the client configuration, how the enforcement hierarchy is actually structured, how to test that a cap really fires before you rely on it in production, and how to decide between hard- stopping a request and degrading it to a cheaper model.

Why RPM limits don’t govern spend

Classic rate limiting counts requests inside a time window. That model assumes requests cost roughly the same, which is true for CRUD endpoints and badly false for LLM traffic. One chat completion with a 200-token prompt and one that stuffs 60,000 tokens of retrieved context into the window both tick the counter by exactly one, while differing by two orders of magnitude in what they cost you.

So a per-minute cap can be fully respected while the monthly bill triples. If you want to control spend, you have to meter the things that generate spend: tokens consumed and dollars charged. We cover the underlying reasoning in more depth in token-based rate limiting for AI agents and in the broader guide to API cost protection with rate limits, quotas, and spending caps.

The gateway pattern below is what makes that metering possible without asking every team to instrument their own code.

What changes in your client

Exactly two values change: the base URL and the API key. Your request payloads stay identical — same messages array, same temperature and tool definitions, same streaming behavior. Zuplo’s Universal API follows the OpenAI API specification, so anything that speaks OpenAI already speaks to it.

The endpoint is your AI Gateway app’s API URL with /v1 appended, because client libraries expect a base URL ending in /v1. The app URL includes the app’s config ID:

https://my-gateway-main-2e18f50.zuplo.app/config_fe0a04972d2848e0a94ae4b8bcd1497e/v1

Two mechanisms can tell the gateway which app is calling, and it’s worth being precise about which one is in play. When the app’s chain includes the API Key Authentication policy, the app is resolved from the key you send as a bearer token. Without that policy, the gateway falls back to the app ID segment in the URL. Most production setups use the key, because a URL is not a secret.

One naming convention matters. In the Universal API, models are always named providerName/model, where providerName is the provider you configured in your gateway. So it’s openai/gpt-5-mini, not gpt-5-mini. The prefix is what routes the request, which is exactly why a single app can reach models from several providers through the same endpoint.

Beyond chat completions, the Universal API also exposes /v1/embeddings (every provider except Anthropic), /v1/responses (OpenAI and Bedrock Mantle models that serve it), and Anthropic’s /v1/messages shape (Anthropic and Bedrock Mantle Claude models).

Step 1: Point your client at the gateway

Take your existing client construction and change the two values. Everything below the constructor — the model call, the response handling — stays exactly as it was. Read past the snippet before you copy it; the two annotations that follow are where the mistakes happen.

// client.ts — same SDK, same request shape, different base URL
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ZUPLO_AI_GATEWAY_API_KEY,
  baseURL:
    "https://my-gateway-main-2e18f50.zuplo.app/config_fe0a04972d2848e0a94ae4b8bcd1497e/v1",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5-mini",
  messages: [{ role: "user", content: "Summarize our Q1 sales data." }],
});

console.log(response.choices[0].message.content);

baseURL points at your gateway, not api.openai.com. The gateway proxies to OpenAI using the provider key stored in your gateway configuration, so the raw provider key never has to live in the application environment. That alone removes a whole category of credential-sprawl problems.

apiKey is the gateway app’s key, not your OpenAI key. This is how the gateway identifies the caller and resolves which team and app the request belongs to, which in turn determines which limits apply. Pasting a real OpenAI key here is the single most common mistake with this setup — it defeats the identity model the entire governance layer depends on.

Python takes the same two options under snake-case names. Zuplo documents the TypeScript path, but the Python SDK accepts an OpenAI-compatible base URL the same way:

# client.py
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://my-gateway-main-2e18f50.zuplo.app/config_fe0a04972d2848e0a94ae4b8bcd1497e/v1",
    api_key=os.environ["ZUPLO_AI_GATEWAY_API_KEY"],
)

response = client.chat.completions.create(
    model="openai/gpt-5-mini",
    messages=[{"role": "user", "content": "Summarize our Q1 sales data."}],
)

print(response.choices[0].message.content)

Anything else that accepts an OpenAI-compatible base URL works the same way — LangChain, the AI SDK, and agent-style coding tools included.

The enforcement hierarchy

Once traffic flows through the gateway, you configure limits at three levels. Zuplo’s usage limits apply at the gateway, team, and app tiers:

• Gateway — project-wide limits under Settings → Usage Limits, applying across every team and app.
• Team — configured on the team’s Usage & Limits tab, covering all apps in that team.
• App — configured through the Budgets and Costs policy in the app’s policy chain.

At every level you get three independent meters, each with its own daily and monthly settings:

• Costs — spend in US dollars. The portal labels this meter Budget; costs is the key you use in policy configuration.
• Tokens — input plus output tokens combined.
• Requests — plain request count, for when you do want throughput control.

The important detail is how the tiers combine. All three apply together, and a request is blocked as soon as any one level’s limit is exceeded. The gateway checks the app, then its team, then the gateway itself, and stops at the first level that has been exceeded. Team and gateway limits are enforced outside the app’s policy chain, so they hold whether or not an individual app has the Budgets and Costs policy configured — a team can’t opt out of its ceiling by removing a policy.

One scheduling nuance that surprises people: daily and monthly periods are anchored to the gateway’s creation time in UTC, not to the calendar. Changing a limit mid-period does not reset accumulated usage.

Step 2: Configure the app-level policy

App limits come from the Budgets and Costs policy, whose policy type is ai-gateway-metering-v2-inbound. Configured directly in policies.json it looks like this:

{
  "name": "ai-gateway-metering-v2-inbound",
  "policyType": "ai-gateway-metering-v2-inbound",
  "handler": {
    "export": "AIGatewayMeteringV2InboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "throwOnFailure": false,
      "limits": {
        "costs": {
          "monthly": {
            "enabled": true,
            "limit": 100,
            "warning": { "enabled": true, "threshold": 80 }
          }
        },
        "requests": {
          "daily": { "enabled": true, "limit": 1000 }
        }
      }
    }
  }
}

Three behaviors are worth internalizing before you ship this.

Every period has to be enabled explicitly

A period is enforced only when enabled is true and a limit is present. Options that set only limit are silently inert. That’s a footgun if you assume you’re protected, but it’s also genuinely useful: it gives you a clean way to stage a rollout, observing real spend against a proposed number before you turn enforcement on.

Warning thresholds fire without blocking

The warning.threshold value is a percentage of the limit. At 80% of a $100 monthly budget, a warning event is emitted and traffic continues. Teams get advance notice rather than discovering the cap when their product starts returning errors.

Fail-open versus fail-closed

throwOnFailure defaults to false, which means the policy fails open — if the metering check itself errors, requests are allowed through. Setting it to true closes that gap, but only partially, and the limitation is important: the flag governs failures while checking or recording the app’s own limits. It does not change inherited-limit behavior. If the central hierarchical check is unavailable, the request still proceeds. So throwOnFailure: true tightens app enforcement without making team and gateway ceilings fail closed.

Policy order in the chain matters too. The documented recommendation is Model Filtering → Fallback Model → Budgets and Costs → Semantic Cache. Budgets runs after the model is resolved (you can’t price a request before you know the model) and before Semantic Cache, so that cache hits still count toward request limits.

Step 3: Verify the cap fires

Don’t take enforcement on faith. Before you depend on it, prove it in a scratch environment.

Create a test app, set a deliberately low daily costs limit, and make sure enabled is true. Policy changes apply within about a minute, so wait before testing rather than firing immediately after saving. Then run the snippet above enough times to cross the threshold, and confirm the next call is rejected.

When a limit is exceeded, the gateway blocks the request with an HTTP 429 Too Many Requests, and the error names the level (app, team, or gateway), the meter (costs, tokens, or requests), and the period (daily or monthly) that blocked it. That specificity is what makes the failure actionable: “app monthly costs limit” tells an on-call engineer something, where a bare 429 does not.

On the client side, treat this as a distinct failure mode from a provider-side rate limit. One practical gotcha: the OpenAI SDKs retry 429s automatically — two retries by default with short exponential backoff — which is exactly wrong against an exhausted budget, since the window won’t reset for hours or days. Set maxRetries: 0 (Node) or max_retries=0 (Python) on requests you route through the gateway, catch the 429 yourself, and log the level and meter so your dashboards can attribute it. For user-facing surfaces, translate it into something like “usage limit reached for your team” instead of leaking raw gateway errors. Our guide to HTTP 429 handling covers backoff strategy in more detail.

Hard stop or fall back?

A 429 is the default, and for strict budget guarantees it’s the correct behavior — nothing slips through once the cap is hit.

But blocking isn’t the only option. If the app’s policy chain includes an AI Gateway Fallback Model supplying a quota fallback, an exceeded limit routes the request to that cheaper model instead of returning a 429. Users keep getting responses, just from a less expensive model. Note that the fallback’s usage still counts toward the same limits, so this degrades cost rather than ignoring it.

Choosing between them comes down to which failure you’d rather explain:

• Hard stop (429) when finance has approved a fixed envelope and an overage is unacceptable. Best for internal tools, batch jobs, and anything where a failed request is merely inconvenient.
• Quota fallback when availability outranks precision. Best for user-facing features where a slightly worse answer beats an error page.

Both are policy configuration. Neither requires an application code change, which means you can start with hard stops while you learn your real usage patterns and relax specific apps to fallback later.

What centralization unlocks

The reason this pattern is worth the setup cost is that the baseURL change is a one-time investment that keeps paying out. Once every LLM call flows through one place, other controls become configuration rather than engineering projects:

• Multi-provider routing. OpenAI, Anthropic, Google, Mistral, xAI, Amazon Bedrock (via Bedrock Mantle, the compatible-APIs endpoint rather than native InvokeModel), and OpenAI-compatible custom providers can all sit behind the same endpoint, selected by the model name prefix.
• Semantic caching. The AI Gateway Semantic Cache policy matches on vector similarity rather than exact string equality, so near-identical prompts can be served from cache. This is an enterprise feature, free to try on any plan for development.
• Per-team attribution. Because each app key resolves to a team, you can finally answer which workload drove the bill instead of staring at one undifferentiated provider invoice.
• Provider key custody. Keys live in the gateway, not in a dozen application environments and CI secrets.

That last point is worth being honest about, though. A gateway is an enforcement point, not a fence. If a developer can still open a socket to api.openai.com with a personal key, nothing in the policy layer stops them. Closing that path is a network egress and IAM problem: restrict outbound traffic to provider domains, and strip provider permissions from individual developer identities so only the gateway’s credentials can reach them. The gateway makes the sanctioned path the easy one; your network controls make the unsanctioned path hard.

Where to go next

If you’re evaluating whether to centralize AI traffic at all, start with AI governance for API teams for the organizational framing, or the AI gateway buyer’s guide if you’re comparing options. For the attribution side — turning gateway telemetry into per-team cost reporting — see attributing AI spend to teams and apps.

If you’re ready to build, the practical sequence is short: create a gateway, add your provider keys, create a team and an app, copy the app’s API URL, append /v1, and change one line in your client.