Load Testing Through Proxies, Gateways, and CDNs

Almost no production API is reachable directly. A request usually crosses DNS, a CDN, a WAF, a load balancer, and an API gateway before the origin sees it. A load test measures all of them, and it tends to find the weakest layer first.

That is exactly why you should test the production path. It is also why blaming the origin, or the gateway, from one end-to-end number is mostly guesswork.

Write down the expected cost of every layer

Before the test, document each hop and the latency it is expected to add. Include the conditions behind the number: region, policies, cache state, connection reuse, and whether the node is warm.

We publish our own numbers. A Zuplo gateway adds about 20 to 30ms of base latency with no policies. Most policies add 1 to 5ms, while policies that call an external service generally add 5 to 15ms. A cold request to a fresh edge node can be 100 to 200ms slower. Our performance-testing guide documents the baseline and testing conditions.

Those are planning numbers, not universal constants. Get the equivalent from every provider in your path, then add the expected costs into one end-to-end budget.

This budget does not prove which layer caused an outlier. It tells you whether the end-to-end result is plausible and gives each owner a number to validate. Without it, a latency chart has no baseline.

Warm the path before collecting steady-state data

Edge and serverless platforms start with little or no capacity in a cold location. The first requests can pay for runtime initialization, configuration loading, and connection setup. A test that jumps from zero to peak traffic mixes startup behavior into its steady-state percentiles.

Use a separate warm-up scenario and exclude its samples from thresholds. The following k6 configuration sends 20 RPS for two minutes before starting the 500-RPS measurement window:

export const options = {
  discardResponseBodies: true,
  scenarios: {
    warmup: {
      executor: "constant-arrival-rate",
      rate: 20,
      timeUnit: "1s",
      duration: "2m",
      preAllocatedVUs: 20,
      startTime: "0s",
    },
    measurement: {
      executor: "constant-arrival-rate",
      rate: 500,
      timeUnit: "1s",
      duration: "5m",
      preAllocatedVUs: 200,
      maxVUs: 500,
      startTime: "2m",
    },
  },
};

k6 tags metrics with the scenario name. Use that tag in every threshold so the warm-up data stays out of the measurement.

In our own seven-minute runs against a trivial origin, the gateway added about 40ms end to end after the edge had warmed. The first couple of minutes were higher while nodes absorbed the step-function ramp. That gap matters: the 40ms includes the extra network hop, while the published base number measures gateway processing. Mixing those two numbers would create an argument where there is no disagreement.

Warm-up is not permission to hide cold starts. Measure cold-start behavior in a separate test if it matters to your users. Keep the two questions separate: steady-state capacity and latency after a scale-to-zero event are different properties.

Treat expected 429 responses as expected

A rate-limit test should cross the configured limit. If it never receives a 429, the limiter may not be enforcing the policy. If the script counts every 429 as an unexpected failure, a working limiter makes the test fail.

In k6, declare 429 as an expected response and count it separately:

import http from "k6/http";
import { Counter } from "k6/metrics";

const BASE_URL = __ENV.TARGET_URL || "https://api.example.com";

http.setResponseCallback(http.expectedStatuses({ min: 200, max: 299 }, 429));

const rateLimited = new Counter("rate_limited");

export default function () {
  const res = http.get(`${BASE_URL}/products`);
  if (res.status === 429) {
    rateLimited.add(1);
  }
}

Add scenario-specific thresholds to the options object:

thresholds: {
  "http_req_failed{scenario:measurement}": ["rate<0.01"],
  "rate_limited{scenario:measurement}": ["count>0"],
  "http_req_duration{scenario:measurement}": ["p(95)<300", "p(99)<800"],
},

The callback keeps expected 429s out of http_req_failed. The custom counter makes the run fail if the limiter never activates.

429s still contribute to http_req_duration, and they are often faster than successful responses. Keep the status-specific latency trends from the first article so quick rejections do not improve the reported latency.

For Zuplo’s inbound rate-limit policy, the configured mode changes the request-path cost:

Use strict counting when the limit protects billing or contractual usage. An asynchronous limit can be appropriate when the goal is protecting infrastructure and a small overshoot is acceptable.

This is not theoretical for us. A sustained load test exposed an unhandled exception in our rate-limiter client when its backing service responded slowly. Our functional tests had never driven that error path hard enough to find it. We fixed the client, but the more useful lesson was that a full-path load test audits the defensive machinery too. Testing around the limiter would have missed the bug entirely.

Avoid sub-second windows for a distributed rate limiter. At that scale, the test can become a measurement of synchronization delay rather than the policy you intended to validate.

Define the cache hit ratio in the test design

Every request through a cache is either a hit or a miss. The ratio changes latency, origin load, intermediary work, and often rate-limit behavior.

A script that repeatedly requests one URL usually creates a nearly 100% hit ratio after the first request. That may be useful for measuring cached delivery, but it tells you little about origin capacity. Randomizing every key creates the opposite problem.

Choose keys from a production-like distribution. Report the measured hit ratio next to the latency and throughput results. If you need both extremes, run separate cache-hot and cache-cold scenarios instead of blending them.

Map every hop and keep evidence from every layer

Draw the full request path before the first request. Include components that are easy to overlook:

• DNS proxies and secondary CDN accounts
• WAF and bot-management services
• Public and private load balancers
• NAT gateways and egress proxies
• API gateways and service-mesh proxies
• The origin platform in front of application code

For each layer, record request count, status, latency, and request or trace ID when available. If one layer logs a request and the next layer does not, you have narrowed the loss to one segment of the path.

Do not tear down test infrastructure immediately after the run. Keep the generators, dashboards, logs, and target environment available until the team has finished investigating. A failed run without its evidence is just an expensive retry.

Providers may also require advance notice. DDoS and bot systems are designed to block high request rates from a small set of IPs. Review each provider’s policy, book the test window, and allowlist generator addresses. Akamai publishes load-testing guidance, and AWS distinguishes permitted testing from simulated DDoS events.

Make the origin behave like the real service

A hello-world origin hides concurrency limits. Little’s law says that in-flight requests equal arrival rate multiplied by average response time. At 30,000 RPS, a 400ms origin creates roughly 12,000 in-flight requests. A 5ms mock creates about 150.

Those workloads put very different pressure on connection pools, load balancers, gateways, and NAT tables.

Give the test origin a configurable delay and set it from production latency, or replay a realistic response-time distribution. Add a degraded scenario with multi-second responses to expose interactions between timeouts, retries, and queue shedding.

Full-path load test checklist

Before the run:

1. Draw every hop from generator to origin.
2. Record the expected overhead and operating conditions for each layer.
3. Confirm the provider test window and allowlist generator IPs.
4. Add a warm-up phase and exclude it from steady-state thresholds.
5. Define expected status codes, including 429s.
6. Define and measure the cache hit ratio.
7. Use an origin with realistic latency and failure behavior.
8. Keep the environment and telemetry available for investigation.

A full-path test should measure the full stack. We publish our overhead so you can hold us to it, and we would rather help set up a clean test than argue about a dirty result afterward. When the result is wrong, the next job is attribution. The following article shows how to use logs and distributed traces to find the responsible layer.