Load Testing Through Proxies, Gateways, and CDNs

One customer spent a long time chasing a load test where a small but consistent fraction of requests just hung. Not slow, not errored - hung, indefinitely, with no log line anywhere obvious to explain it. Their API was fine under the same load without the test harness. The gateway looked fine. The origin looked fine.

The answer turned out to be a hop nobody had put on the diagram. They had a DNS proxy sitting in front of their gateway, and both layers happened to run on the same underlying edge platform through two separate accounts. Requests were vanishing at the boundary between them. The team built a clean repro - same test with the extra proxy layer, hangs on every run; same test without it, none - and removing the layer fixed it. The request IDs logged at the outer layer never appeared at the inner one, which is exactly what a request disappearing between two hops looks like in the data.

That is the shape of most confusing load test results. Almost no production API is reachable directly. Between a load generator and the origin (i.e. the backend that does the work) you’ll usually find a CDN, a WAF (web application firewall), a load balancer, an API gateway, and sometimes several of each. A test through that path measures every one of those intermediaries plus your API, and it finds the weakest one first - which is almost never the one you set out to test.

Testing through the full path is still the right call, because that’s the path production traffic takes. The mistake is testing through it as if it weren’t there. Those layers add latency, cache responses, enforce rate limits, pool connections, and sometimes carry terms of service that treat an unannounced test as an attack. The last post covered where load generators should run; this one takes the path between them a layer at a time.

Budgeting each layer’s overhead

Get a number from every layer in the path before you test: how much latency it adds per request, and the conditions under which that holds. We publish ours. Zuplo adds approximately 20-30ms of base latency with no policies, most policies add 1-5ms each, and policies that make external calls (authentication, rate limiting, or custom code calling another service) add 5-15ms. The caveat is cold starts: a first request to a fresh edge node may be 100-200ms slower, enough to shape the whole test design.

Ours are one example, and you want the equivalent in writing from everyone else, because those numbers add up to what the whole path should cost before you send a single request. A result inside that total means the layers are behaving and any remaining problem is likely the origin or the network. A result well outside it points at one layer, and you know which numbers to go argue about. Without it, every measurement is just a number with nothing to compare it against. A vendor who can’t tell you what their own product costs has told you something before the test even starts.

Warming up the path

Serverless and edge platforms scale capacity to match demand, and demand at the start of a load test is zero. The first request to a new node pays for initialization, so a test jumping from nothing to thousands of RPS records cold-start latency as steady-state behavior.

Cold starts also show up backwards, which throws people. One customer came to us with p99 latency spiking overnight, when their traffic was at its lowest, and smoothing out once daytime traffic reached its plateau. The graph looked like their API got slower the less work it had to do. It did, in a sense: at low volume nearly every request lands on a node that has gone cold since the last one. A latency graph inverted relative to traffic is almost always cold starts, not load.

Initialization cost scales with configuration size too, and that one can get dramatic. One customer’s API had over a thousand routes with complex validation schemas attached, and new nodes initialized slowly enough that even 404s were taking multiple seconds. A 404 is about the cheapest response an API can produce. If yours takes seconds at low traffic, you are almost certainly timing node startup rather than request processing.

Synthetic tests exaggerate all of this: a load test’s instant step lands on one or two locations, while real traffic typically ramps more linearly across hundreds of edge locations, so capacity warms ahead of it. We’ve measured the ideal case: generator near the backend, trivial origin, roughly seven-minute runs. The TLDR is that the gateway added about 40ms end to end, with the first couple of minutes elevated as edge nodes scaled into the step-function ramp. The 40ms exceeds the base-latency figure above because it covers the whole added hop from the client, network included, not gateway processing alone. That window is an artifact of the test shape and also real: send a few hundred to a thousand requests before the measurement window opens.

In k6 that’s two scenarios, warm-up and measurement, on the constant-arrival-rate executor so the open workload model (requests arrive on a schedule regardless of responses) carries over from the first post:

export const options = {
  // Only the status code gets read, so don't pay to buffer bodies at 500 RPS.
  discardResponseBodies: true,
  scenarios: {
    warmup: {
      executor: "constant-arrival-rate",
      rate: 20,
      timeUnit: "1s",
      duration: "2m",
      preAllocatedVUs: 20,
      startTime: "0s",
    },
    measurement: {
      executor: "constant-arrival-rate",
      rate: 500,
      timeUnit: "1s",
      duration: "5m",
      preAllocatedVUs: 200,
      maxVUs: 500,
      startTime: "2m", // picks up right where warmup leaves off
    },
  },
};

The warmup scenario trickles 20 RPS for two minutes, and startTime holds measurement back until it finishes. preAllocatedVUs and maxVUs size the pool of VUs (i.e. k6’s virtual users) carrying those arrival rates. k6 tags every metric with the scenario name, so warm-up samples stay out of the thresholds and reports - every threshold in the next section does that.

Designing tests that expect 429s

If there’s a rate limit anywhere in the path, a serious load test will almost certainly cross it. A test that counts the resulting 429s as failures ends with someone debugging an outage that never happened. A correct test asserts that 429s happen: push past the configured limit without seeing them and the limit isn’t enforcing, which is a real finding.

In k6, the http.setResponseCallback API declares which statuses count as expected:

import http from "k6/http";
import { Counter } from "k6/metrics";

const BASE_URL = __ENV.TARGET_URL || "https://api.example.com";

// A 429 is a correct response for a test built to cross a rate limit.
http.setResponseCallback(http.expectedStatuses({ min: 200, max: 299 }, 429));

const rateLimited = new Counter("rate_limited");

export default function () {
  const res = http.get(`${BASE_URL}/products`);
  if (res.status === 429) {
    rateLimited.add(1);
  }
}

With the callback in place, 429s stop counting toward http_req_failed, which stays reserved for real failures. The rate_limited counter lets the thresholds assert both directions at once:

thresholds: {
  // Real failures only, since 429s are expected responses here.
  "http_req_failed{scenario:measurement}": ["rate<0.01"],
  // The test is meant to cross the limit, so fail the run if it never did.
  "rate_limited{scenario:measurement}": ["count>0"],
  // Warmup cold starts stay out of the percentiles.
  "http_req_duration{scenario:measurement}": ["p(95)<300", "p(99)<800"],
},

The snippets assemble into one file: imports and callback at the top, then options, then the default function. 429s still land in http_req_duration like any other response, and they tend to be fast. If the test spends much of its time over the limit, keep the per-status latency segmentation from the first post in this series so quick rejections don’t flatter your percentiles.

Rate limiting has its own line in that budget. In Zuplo’s rate limit policy, mode decides whether the check sits on the request path:

Pick based on what the limit protects: revenue (billing enforcement) needs strict counting; infrastructure is usually fine with async.

Load tests audit this machinery in ways functional tests never will. A customer’s load test is what exposed an unhandled exception in our own rate-limiter client under slow responses: their test found a bug in our product that our own tests hadn’t, we shipped the fix, and everyone’s limiter got better for it. That is a good argument for running these tests against the real path rather than around it.

Don’t use sub-second rate-limit windows, in tests or anywhere else. Distributed rate limiters typically synchronize counts across nodes at intervals comparable to a sub-second window, so enforcement gets noisy and the test measures the synchronization, not the policy.

Stating your cache hit ratio

Any test through a caching layer has a cache hit ratio (i.e. the fraction of requests answered from cache instead of the origin), whether you chose one or not. The same endpoint at the same RPS produces different latency distributions at a 0% hit ratio and a 95% one, so a result without its hit ratio is neither reproducible nor comparable.

The default script most people start from hits one URL with one key, unrealistically fast and unrealistically throttled at once. A single hot key never misses cache after the first request, so the origin and most of each intermediary’s work vanish from the measurement. Every request also lands in the same rate-limit bucket and cache node. Draw request keys from a production-like distribution, decide what hit ratio you’re simulating, and report it with the latency numbers. Our guide to solving latency problems in high-traffic APIs catalogs the caching and edge options themselves.

Mapping the topology

Before the first request, draw every hop between the generator and the origin. Most teams know about the gateway and the CDN; the troublesome hops tend to be the ones nobody remembers are there - a DNS proxy in front of everything, a NAT gateway on the egress side, a second account at a provider already in the path.

The hanging-requests story at the top of this post is exactly that: a DNS proxy nobody thought of as a hop, sharing an edge platform with the gateway behind it through a second account. Two lessons generalize from it. Stacked layers from one provider make their shared boundary the first place to look. And a timeout with no matching entry in the next layer’s logs tells you precisely which two hops it vanished between, which is usually faster than any amount of staring at either end.

The weakest layer also tends to be an unglamorous one. During one long test campaign we watched, run after run kept topping out at a rate nobody could explain. The cause was a managed load balancer in front of the origin with a hard per-instance ceiling on concurrent connections, far below anything the gateway would ever have hit. The real cost wasn’t finding it, though - it was that nobody could dig into any of those runs afterward, because the test clusters had already been torn down each time. Leave the infrastructure up until everyone’s forensics are done. It is the cheapest line item in the whole exercise. When a layer comes under suspicion, our troubleshooting guide walks through isolating it.

The map also shows which layers might block the test outright. At scale, every automated defense in the path sees your test as an attack: DDoS mitigation and bot rules exist to stop exactly the shape it produces, very high RPS from a few IPs with near-identical fingerprints. Teams worry about this enough to ask the vendor directly. One major CDN’s published load-testing policy wants a support ticket at least 24 hours before every test; AWS’s testing policy draws an explicit line between network stress tests and simulated attacks. Unannounced tests often get fingerprinted and blocked, and the dataset benchmarks the block page instead of your API.

Simulating a real origin

The origin is the one layer you fully control, and in most setups it’s a hello-world endpoint answering in single-digit milliseconds, which hides most concurrency effects in the path. Little’s law: in-flight requests equal arrival rate times response time. At 30,000 RPS, a 400ms origin means roughly 12,000 requests in flight at any moment; a 5ms mock means about 150. The two hold open wildly different numbers of connections at every intermediary, so a ceiling like the load balancer’s above is invisible in one and the first thing the other finds.

Give the test origin a configurable delay (a ?delay=400ms parameter or a sleep in the handler) and set it from your production response-time distribution, or at least its median. Test the degraded case too: set the delay to a few seconds and watch the intermediaries when the origin crawls. Timeouts, retries, and queue-shedding across layers interact in ways better discovered in a test than in an incident.

Draw the map before you run the test

Everything in this post comes back to the same habit: know what is in the path before you send traffic through it. The customer with the hanging requests spent real time on a mystery that a hop diagram would have made obvious. The team hitting the load balancer’s connection ceiling lost several runs’ worth of evidence to a teardown script. The overnight p99 spike was a graph read backwards.

Before your next full-path test, get five things on paper:

• Every hop between the generator and the origin, including the ones nobody thinks of as hops - DNS proxies, NAT gateways, a second account at a provider already in the path.
• A stated per-request overhead for each layer, in writing, from whoever owns it. Ours is published. A vendor who can’t state one has told you something before the test even starts.
• A warm-up phase, so you’re measuring steady state instead of node startup.
• What the rate limits are and that you expect to cross them, with 429s asserted rather than counted as failures.
• The cache hit ratio you’re simulating, reported next to every latency number.

A full-path test measures the whole stack, which is exactly what your users experience. When a confusing result shows up, the first thing it found is usually the test itself. When the errors are real, the next question is which layer produced them, and the load tool on its own cannot answer that. That’s the next post.

If any part of that path is a Zuplo gateway, come talk to us before the run. We publish our overhead numbers specifically so you can hold us to them, and we’d much rather help you get a clean measurement than argue about a dirty one afterward.