Load Testing Your API from Multiple Regions

A large customer came to us with numbers showing our gateway adding tens of milliseconds to every request. They had done the responsible thing and built a direct-vs-gateway comparison: one leg straight to the backend, one leg through the gateway, same payloads, same auth, same everything. The gateway leg was clearly slower. It looked like an open-and-shut case against us.

The problem was in a setting nobody had touched. They were driving both legs from a hosted load-testing platform whose default region happened to sit in the same cloud as their backend. So the direct leg never left the provider’s private network, while the gateway leg paid real public-internet costs on every request, out and back. We re-ran the same comparison from neutral bare metal in a third location and the measured worst-case overhead roughly halved. The gateway had not changed; the baseline had been measuring a private network path the whole time.

Nothing about that setup is unusual. It is the default one, and most hosted platforms keep their load zones in one or two clouds - every public Grafana Cloud k6 zone runs on AWS - so if your backend is on AWS, the odds are good that your baseline is already rigged and nobody decided to rig it.

Where your load generators run (i.e. the machines sending the requests) shapes just about every number a load test produces, and in most tests nobody chose that location at all. The tool picked a default region, or someone grabbed the VM nearest the office, and every latency figure in the report inherited that choice. The previous post in this series covered making the numbers mean something. This one covers the physical half - where the requests actually come from.

Nothing in the tooling flags this. Generator placement doesn’t appear in the script, the report, or the dashboards, even though it shapes all three. Let’s go through the placement decisions in order.

Running load generators on a different cloud than your backend

Cloud providers route traffic between their own data centers over private backbone, and traffic between services in one region often never leaves the building. A generator sharing a provider with your backend measures a path almost none of your users will take. The absolute numbers come out flattering, but the variance gives it away: public-internet latency has jitter, occasional retransmits, and a visible spread between p50 and p99 (we covered reading percentile spread instead of averages previously), while an intra-cloud path is implausibly flat.

The remedy costs almost nothing: as our performance-testing guide puts it, never run performance tests from the same cloud provider as your backend, and prefer regions where your users are. For an A/B comparison to be fair, run both legs over the public internet from the same locations with identical payloads and auth, so the only variable left is what you’re comparing.

Spreading load across regions and resolvers

One region of generators creates a second, less obvious problem once the target is multi-region. A DNS-based global load balancer typically picks a data center from where the client’s resolver sits, not where the client sits, so each generator location resolves once and lands on one data center. Run it from a few locations and you have tested one cluster with extra DNS noise on top.

Anycast-fronted systems get there differently: many locations advertise the same IP address and routers direct each request to the nearest machine, so a given generator location lands on the same PoP (point of presence) essentially every time, and one city’s test measures one PoP’s ceiling out of hundreds.

From the inside this looks exactly like a backend capacity problem, which is why it wastes so much time. A customer of ours was validating a brand new two-region, active-active deployment behind a DNS-based global load balancer. They fired a few thousand RPS at it from a couple of hosts near one metro. Instances started running out of memory and restarting, error rates climbed, and the conclusion in the room was that the new architecture couldn’t take the load.

The second region was idle the entire time. Between DNS caching, connection reuse, and location-based routing, every single request had gone to one region - a region provisioned for its share of the traffic, not all of it. The deployment had been failed for a test it was never actually given.

The tell was sitting in the DNS data. A multi-thousand-RPS test had produced only a few hundred lookups, which is the signature of a handful of clients resolving once and then hammering the one answer they got. Query volume tells you where your load actually went, and most managed DNS providers expose counts per zone and per record. Check it on every distributed test - it is one graph and it settles the argument.

The saturation reached the telemetry too, in a way worth recognizing. The load tool reported roughly twice as many successful responses as the gateway’s analytics did. Crashed instances explain the gap: an instance that dies never flushes its telemetry, so the tool had correctly counted successes that no instance survived to report. The error counts disagreed for a separate reason - 5xx responses from the load balancer in front of the gateway never appear in gateway analytics at all. Observability tends to degrade first under saturation, so when two layers stop agreeing on the numbers, that disagreement is itself the finding.

What to do depends on the question. For realism (production-shaped traffic), spread generators across regions and resolvers so the load balancer sees a population like your users. For regional capacity, do the opposite: bypass the global load balancer and hit each region at the share it’s provisioned for. In k6 that’s one script with two scenarios, each pinned to a region’s direct endpoint, driven by an arrival-rate executor (an open workload model, where requests arrive on a schedule regardless of response times; the previous post covers why that matters). Each scenario pre-allocates its own pool of VUs (i.e. virtual users, k6’s request-sending workers) and grows it when responses slow:

import http from "k6/http";

// Bypass the DNS load balancer: point each scenario at one region's
// direct endpoint, each carrying that region's share of the total load.
const REGION_A = __ENV.REGION_A_URL || "https://region-a.api.example.com";
const REGION_B = __ENV.REGION_B_URL || "https://region-b.api.example.com";

export const options = {
  // we only need the timings, and bodies at these rates burn generator memory
  discardResponseBodies: true,
  // each scenario tags its own metrics, so you can tell the regions apart
  scenarios: {
    region_a: {
      executor: "constant-arrival-rate",
      exec: "regionA",
      rate: 1000,
      timeUnit: "1s",
      duration: "10m",
      preAllocatedVUs: 200,
      maxVUs: 500, // separate pool per scenario, so budget for 1,000 VUs
    },
    region_b: {
      executor: "constant-arrival-rate",
      exec: "regionB",
      rate: 1000,
      timeUnit: "1s",
      duration: "10m",
      preAllocatedVUs: 200,
      maxVUs: 500,
    },
  },
};

export function regionA() {
  http.get(`${REGION_A}/v1/resource`);
}

export function regionB() {
  http.get(`${REGION_B}/v1/resource`);
}

On a hosted platform, geographic spread is typically a first-class option: Grafana Cloud k6 distributes a test across zones with a cloud.distribution block in the same options object:

export const options = {
  // ...the scenarios block from above...
  cloud: {
    distribution: {
      us: { loadZone: "amazon:us:ashburn", percent: 50 },
      asia: { loadZone: "amazon:jp:tokyo", percent: 50 },
    },
  },
};

Check where a platform’s zones physically run before trusting a baseline: a zone sharing a cloud provider with your backend brings back the earlier problem.

Sending load from more than one source IP

Spreading across regions usually handles source diversity, but treat IP count as its own requirement: several systems in the path key on client IP. Rate limiting is the obvious one: under per-IP limits at the API or the gateway, two generator hosts look like two wildly abusive clients instead of thousands of normal ones, and you measure the rate limiter rather than the API. WAF and bot-mitigation rules score by IP too, and one address at thousands of RPS is the exact profile they exist to stop.

Load balancers add a subtler version. Many of them pool connections from a small set of source IPs, so a two-host test can be funneled through less of the load balancer than production traffic would. Source-IP hashing (affinity schemes where the same client always reaches the same backend) does the same one layer down, concentrating a two-IP test onto one or two backend nodes, so your numbers describe those nodes, not the fleet.

The fix is more generator hosts, each with its own address, spread across those regions. Hosted platforms typically fan out over many machines and IPs, but confirm rather than assume. And if a WAF or bot-mitigation layer sits in the path, tell its operator a test is coming so their automation doesn’t decide for you.

Monitoring the load generator as part of the system under test

Load generators tend to fail quietly. An overloaded backend throws errors; an overloaded generator sends less load than you asked for, still produces a normal-looking report, and its numbers drift optimistic because fewer requests in flight means less queueing at the target. The fleet has CPU, memory, file descriptors, an ephemeral port range, and a network path of its own, any of which can become the run’s limiting factor.

The one that catches people out is the connection budget. The TLDR is that a generator host can only hold so many connections open at once, and once you run out, the test starts behaving as though the API were throttling you: requests stall, connections get refused, and throughput flattens at a ceiling that nothing on the server side explains. It is a miserable thing to diagnose precisely because every symptom points at the target.

Two limits produce it. Every outbound connection consumes an ephemeral port, and the default Linux range, 32768 to 60999, provides 28,232 of them; Cloudflare’s write-up on ephemeral port exhaustion has the details. High rates without connection reuse churn through that range, and connections in TIME_WAIT keep ports out of circulation after their requests finish. Separately, anything doing NAT or stateful filtering in front of the generators records every connection in a tracking table of finite size (conntrack on Linux), and once that table is full new connections get dropped rather than queued. An AWS NAT gateway supports up to 55,000 simultaneous connections to each unique destination per IP address, and since a load test is a great many connections to a single destination, a whole fleet behind one NAT gateway shares that one cap.

The practical answer is usually not to solve this yourself. A hosted platform like Grafana Cloud k6 spreads a run across enough machines and addresses that no single host approaches either ceiling, and that distribution is a good part of what you are paying for. If you do run your own generators, add hosts rather than tuning sysctl values on a few of them, and treat an unexplained throughput ceiling as the generator’s fault until you have ruled it out.

How fast you get there depends on connection reuse, and tools differ. Apache’s ab defaults to keep-alive off, opening a fresh connection per request, while k6 reuses connections unless you set noConnectionReuse. Two tools reporting the same RPS can therefore exercise completely different connection loads, so know which yours does before comparing across tools.

The k6 guide to running large tests recommends keeping the generator’s CPU below 80% and memory below 90%, and the reasoning applies to any tool: past that point it distorts timings long before anything visibly fails. k6 also has a built-in tell: dropped_iterations increments whenever an arrival-rate scenario can’t start an iteration on schedule because no VU is free. A nonzero value means the test is no longer sending the load the script describes, so wire it into a threshold and let the run fail itself:

export const options = {
  // ...the scenarios block from above...
  thresholds: {
    dropped_iterations: ["count==0"],
    // timings from a run that was mostly errors aren't worth reading
    http_req_failed: ["rate<0.01"],
  },
};

Headroom in maxVUs keeps that threshold green, since the VUs an arrival rate needs scale with response time; size it for the degraded case you want to observe, not the happy case. If iterations still drop with generous headroom and healthy CPU, that is typically the finding - holding the rate needs more capacity than the fleet has, so add generator machines rather than loosen the threshold.

Decide where your generators run on purpose

Both stories in this post end the same way. A team ran a test, got a bad result, and drew a conclusion about their system - when what they had actually measured was where they happened to be sending traffic from. One team nearly filed a performance regression against a gateway. The other nearly re-architected a deployment that was working.

The checklist that would have caught both is short enough to run before every distributed test:

• Generators in a different cloud than the backend, and both legs of any A/B comparison over the same path.
• More than one region, chosen to match where your users are.
• More than one resolver and source IP, so rate limiters, WAFs, and source-IP hashing see a population rather than two very rude clients.
• DNS query counts checked against the request count after the run.
• Generator CPU, memory, connection limits, and dropped iterations on the same dashboard as everything else.

Placement is only half of the problem, though. Every intermediary in the path is the other half: load balancers, CDNs, WAFs, and gateways all reshape the result before the origin ever sees it, which is the next post in this series.

And if the test you’re planning points at a Zuplo gateway, get in touch before you run it. We can allowlist your generator IPs, tell you which of our regions you’re actually hitting, and watch the run from our side while it happens.