---
title: "Load Testing Through Proxies, Gateways, and CDNs"
description: "A load test through gateways, CDNs, and load balancers measures every layer in the path, not just your API. How to budget each layer's overhead, warm up the path, design for 429s, and map the topology before blaming anything."
canonicalUrl: "https://zuplo.com/blog/2026/08/11/load-testing-gateways-and-cdns"
pageType: "blog"
date: "2026-08-11"
authors: "nate"
tags: "API Gateway, API Best Practices, API Rate Limiting"
image: "https://zuplo.com/og?text=Load%20Testing%20Through%20Proxies%2C%20Gateways%2C%20and%20CDNs"
---
One customer spent a long time chasing a load test where a small but consistent
fraction of requests just hung. Not slow, not errored - hung, indefinitely, with
no log line anywhere obvious to explain it. Their API was fine under the same
load without the test harness. The gateway looked fine. The origin looked fine.
The answer turned out to be a hop nobody had put on the diagram. They had a DNS
proxy sitting in front of their gateway, and both layers happened to run on the
same underlying edge platform through two separate accounts. Requests were
vanishing at the boundary between them. The team built a clean repro - same test
with the extra proxy layer, hangs on every run; same test without it, none - and
removing the layer fixed it. The request IDs logged at the outer layer never
appeared at the inner one, which is exactly what a request disappearing between
two hops looks like in the data.
That is the shape of most confusing load test results. Almost no production API
is reachable directly. Between a load generator and the origin (i.e. the backend
that does the work) you'll usually find a CDN, a WAF (web application firewall),
a load balancer, an API gateway, and sometimes several of each. A test through
that path measures every one of those intermediaries plus your API, and it finds
the weakest one first - which is almost never the one you set out to test.
Testing through the full path is still the right call, because that's the path
production traffic takes. The mistake is testing through it as if it weren't
there. Those layers add latency, cache responses, enforce rate limits, pool
connections, and sometimes carry terms of service that treat an unannounced test
as an attack. The
[last post covered where load generators should run](/blog/load-test-from-multiple-regions);
this one takes the path between them a layer at a time.
## Budgeting each layer's overhead
Get a number from every layer in the path before you test: how much latency it
adds per request, and the conditions under which that holds. We publish ours.
Zuplo adds approximately 20-30ms of base latency with no policies, most policies
add 1-5ms each, and policies that make external calls (authentication, rate
limiting, or custom code calling another service) add 5-15ms. The caveat is cold
starts: a first request to a fresh edge node may be 100-200ms slower, enough to
shape the whole test design.
Ours are one example, and you want the equivalent in writing from everyone else,
because those numbers add up to what the whole path should cost before you send
a single request. A result inside that total means the layers are behaving and
any remaining problem is likely the origin or the network. A result well outside
it points at one layer, and you know which numbers to go argue about. Without
it, every measurement is just a number with nothing to compare it against. A
vendor who can't tell you what their own product costs has told you something
before the test even starts.
## Warming up the path
Serverless and edge platforms scale capacity to match demand, and demand at the
start of a load test is zero. The first request to a new node pays for
initialization, so a test jumping from nothing to thousands of RPS records
cold-start latency as steady-state behavior.
Cold starts also show up backwards, which throws people. One customer came to us
with p99 latency spiking overnight, when their traffic was at its lowest, and
smoothing out once daytime traffic reached its plateau. The graph looked like
their API got slower the less work it had to do. It did, in a sense: at low
volume nearly every request lands on a node that has gone cold since the last
one. A latency graph inverted relative to traffic is almost always cold starts,
not load.
Initialization cost scales with configuration size too, and that one can get
dramatic. One customer's API had over a thousand routes with complex validation
schemas attached, and new nodes initialized slowly enough that even 404s were
taking multiple seconds. A 404 is about the cheapest response an API can
produce. If yours takes seconds at low traffic, you are almost certainly timing
node startup rather than request processing.
Synthetic tests exaggerate all of this: a load test's instant step lands on one
or two locations, while real traffic typically ramps more linearly across
hundreds of edge locations, so capacity warms ahead of it. We've measured the
ideal case: generator near the backend, trivial origin, roughly seven-minute
runs. The TLDR is that the gateway added about 40ms end to end, with the first
couple of minutes elevated as edge nodes scaled into the step-function ramp. The
40ms exceeds the base-latency figure above because it covers the whole added hop
from the client, network included, not gateway processing alone. That window is
an artifact of the test shape and also real: send a few hundred to a thousand
requests before the measurement window opens.
In k6 that's two scenarios, warm-up and measurement, on the
[`constant-arrival-rate` executor](https://grafana.com/docs/k6/latest/using-k6/scenarios/executors/constant-arrival-rate/)
so the open workload model (requests arrive on a schedule regardless of
responses) carries over from the first post:
```javascript
export const options = {
// Only the status code gets read, so don't pay to buffer bodies at 500 RPS.
discardResponseBodies: true,
scenarios: {
warmup: {
executor: "constant-arrival-rate",
rate: 20,
timeUnit: "1s",
duration: "2m",
preAllocatedVUs: 20,
startTime: "0s",
},
measurement: {
executor: "constant-arrival-rate",
rate: 500,
timeUnit: "1s",
duration: "5m",
preAllocatedVUs: 200,
maxVUs: 500,
startTime: "2m", // picks up right where warmup leaves off
},
},
};
```
The `warmup` scenario trickles 20 RPS for two minutes, and `startTime` holds
`measurement` back until it finishes. `preAllocatedVUs` and `maxVUs` size the
pool of VUs (i.e. k6's virtual users) carrying those arrival rates. k6 tags
every metric with the scenario name, so warm-up samples stay out of the
thresholds and reports - every threshold in the next section does that.
## Designing tests that expect 429s
If there's a rate limit anywhere in the path, a serious load test will almost
certainly cross it. A test that counts the resulting 429s as failures ends with
someone debugging an outage that never happened. A correct test asserts that
429s happen: push past the configured limit without seeing them and the limit
isn't enforcing, which is a real finding.
In k6, the
[`http.setResponseCallback`](https://grafana.com/docs/k6/latest/javascript-api/k6-http/set-response-callback/)
API declares which statuses count as expected:
```javascript
import http from "k6/http";
import { Counter } from "k6/metrics";
const BASE_URL = __ENV.TARGET_URL || "https://api.example.com";
// A 429 is a correct response for a test built to cross a rate limit.
http.setResponseCallback(http.expectedStatuses({ min: 200, max: 299 }, 429));
const rateLimited = new Counter("rate_limited");
export default function () {
const res = http.get(`${BASE_URL}/products`);
if (res.status === 429) {
rateLimited.add(1);
}
}
```
With the callback in place, 429s stop counting toward `http_req_failed`, which
stays reserved for real failures. The `rate_limited` counter lets the thresholds
assert both directions at once:
```javascript
thresholds: {
// Real failures only, since 429s are expected responses here.
"http_req_failed{scenario:measurement}": ["rate<0.01"],
// The test is meant to cross the limit, so fail the run if it never did.
"rate_limited{scenario:measurement}": ["count>0"],
// Warmup cold starts stay out of the percentiles.
"http_req_duration{scenario:measurement}": ["p(95)<300", "p(99)<800"],
},
```
The snippets assemble into one file: imports and callback at the top, then
`options`, then the default function. 429s still land in `http_req_duration`
like any other response, and they tend to be fast. If the test spends much of
its time over the limit, keep the per-status latency segmentation from
[the first post in this series](/blog/how-not-to-load-test-an-api) so quick
rejections don't flatter your percentiles.
Rate limiting has its own line in that budget. In
[Zuplo's rate limit policy](/docs/policies/rate-limit-inbound), `mode` decides
whether the check sits on the request path:
| `mode` | Cost on the request path | Trade-off |
| ------------------ | ------------------------------------------------------- | ------------------------------------------- |
| `strict` (default) | Blocks on the external check while the limit is counted | Accurate counts |
| `async` | Approximately 0ms | Some requests can be allowed over the limit |
Pick based on what the limit protects: revenue (billing enforcement) needs
strict counting; infrastructure is usually fine with async.
Load tests audit this machinery in ways functional tests never will. A
customer's load test is what exposed an unhandled exception in our own
rate-limiter client under slow responses: their test found a bug in our product
that our own tests hadn't, we shipped the fix, and everyone's limiter got better
for it. That is a good argument for running these tests against the real path
rather than around it.
Don't use sub-second rate-limit windows, in tests or anywhere else. Distributed
rate limiters typically synchronize counts across nodes at intervals comparable
to a sub-second window, so enforcement gets noisy and the test measures the
synchronization, not the policy.
## Stating your cache hit ratio
Any test through a caching layer has a cache hit ratio (i.e. the fraction of
requests answered from cache instead of the origin), whether you chose one or
not. The same endpoint at the same RPS produces different latency distributions
at a 0% hit ratio and a 95% one, so a result without its hit ratio is neither
reproducible nor comparable.
The default script most people start from hits one URL with one key,
unrealistically fast and unrealistically throttled at once. A single hot key
never misses cache after the first request, so the origin and most of each
intermediary's work vanish from the measurement. Every request also lands in the
same rate-limit bucket and cache node. Draw request keys from a production-like
distribution, decide what hit ratio you're simulating, and report it with the
latency numbers. Our guide to
[solving latency problems in high-traffic APIs](/learning-center/solving-latency-problems-in-high-traffic-apis)
catalogs the caching and edge options themselves.
## Mapping the topology
Before the first request, draw every hop between the generator and the origin.
Most teams know about the gateway and the CDN; the troublesome hops tend to be
the ones nobody remembers are there - a DNS proxy in front of everything, a NAT
gateway on the egress side, a second account at a provider already in the path.
The hanging-requests story at the top of this post is exactly that: a DNS proxy
nobody thought of as a hop, sharing an edge platform with the gateway behind it
through a second account. Two lessons generalize from it. Stacked layers from
one provider make their shared boundary the first place to look. And a timeout
with no matching entry in the next layer's logs tells you precisely which two
hops it vanished between, which is usually faster than any amount of staring at
either end.
The weakest layer also tends to be an unglamorous one. During one long test
campaign we watched, run after run kept topping out at a rate nobody could
explain. The cause was a managed load balancer in front of the origin with a
hard per-instance ceiling on concurrent connections, far below anything the
gateway would ever have hit. The real cost wasn't finding it, though - it was
that nobody could dig into any of those runs afterward, because the test
clusters had already been torn down each time. Leave the infrastructure up until
everyone's forensics are done. It is the cheapest line item in the whole
exercise. When a layer comes under suspicion, our
[troubleshooting guide](/docs/articles/troubleshooting-slow-responses) walks
through isolating it.
The map also shows which layers might block the test outright. At scale, every
automated defense in the path sees your test as an attack:
[DDoS mitigation](https://developers.cloudflare.com/ddos-protection/) and bot
rules exist to stop exactly the shape it produces, very high RPS from a few IPs
with near-identical fingerprints. Teams worry about this enough to
[ask the vendor directly](https://community.cloudflare.com/t/end-to-end-load-testing-with-workers/806638).
One major CDN's
[published load-testing policy](https://community.akamai.com/customers/s/article/Best-Practices-for-Load-Testing-with-Akamai-CDN?language=en_US)
wants a support ticket at least 24 hours before every test;
[AWS's testing policy](https://aws.amazon.com/ec2/testing/) draws an explicit
line between network stress tests and simulated attacks. Unannounced tests often
get fingerprinted and blocked, and the dataset benchmarks the block page instead
of your API.
Allowlist your load generator IPs with every provider in the path before the
first request rather than after the first weird result. It's usually a short
support ticket per vendor, and it turns "is this a WAF or a real failure?"
into a question you never have to ask. Hosted load-testing platforms generally
publish their generator IP ranges for exactly this reason.
## Simulating a real origin
The origin is the one layer you fully control, and in most setups it's a
hello-world endpoint answering in single-digit milliseconds, which hides most
concurrency effects in the path. Little's law: in-flight requests equal arrival
rate times response time. At 30,000 RPS, a 400ms origin means roughly 12,000
requests in flight at any moment; a 5ms mock means about 150. The two hold open
wildly different numbers of connections at every intermediary, so a ceiling like
the load balancer's above is invisible in one and the first thing the other
finds.
Give the test origin a configurable delay (a `?delay=400ms` parameter or a sleep
in the handler) and set it from your production response-time distribution, or
at least its median. Test the degraded case too: set the delay to a few seconds
and watch the intermediaries when the origin crawls. Timeouts, retries, and
queue-shedding across layers interact in ways better discovered in a test than
in an incident.
## Draw the map before you run the test
Everything in this post comes back to the same habit: know what is in the path
before you send traffic through it. The customer with the hanging requests spent
real time on a mystery that a hop diagram would have made obvious. The team
hitting the load balancer's connection ceiling lost several runs' worth of
evidence to a teardown script. The overnight p99 spike was a graph read
backwards.
Before your next full-path test, get five things on paper:
- **Every hop between the generator and the origin**, including the ones nobody
thinks of as hops - DNS proxies, NAT gateways, a second account at a provider
already in the path.
- **A stated per-request overhead for each layer**, in writing, from whoever
owns it. Ours is [published](/docs/articles/performance-testing). A vendor who
can't state one has told you something before the test even starts.
- **A warm-up phase**, so you're measuring steady state instead of node startup.
- **What the rate limits are and that you expect to cross them**, with 429s
asserted rather than counted as failures.
- **The cache hit ratio you're simulating**, reported next to every latency
number.
A full-path test measures the whole stack, which is exactly what your users
experience. When a confusing result shows up, the first thing it found is
usually the test itself. When the errors are real, the next question is which
layer produced them, and the load tool on its own cannot answer that.
[That's the next post](/blog/load-testing-with-tracing).
If any part of that path is a Zuplo gateway, come talk to us before the run. We
publish our overhead numbers specifically so you can hold us to them, and we'd
much rather help you get a clean measurement than argue about a dirty one
afterward.