How Not to Load Test an API

A large customer woke people up over an overnight performance test. A few thousand requests spread across about two hours, one aggregate latency number, and that number was bad enough to page on. As far as anyone on that team could tell, their API had fallen over in the middle of the night.

It hadn’t. We took the same data and grouped it by status code, and three completely different stories fell out of the one number. The 504s had medians sitting right at the 30 second mark. The 503s came back in about five seconds. The successful 200s came back around 280ms, which is what they had been doing all night. Nobody had misconfigured anything and the API was fine. One summary statistic had averaged timeouts, rejections, and successes together into a number that described none of them, and that is the number that got escalated.

Generating load against an API is not the hard part of load testing; any modern tool will happily produce thousands of requests per second from a laptop. In nearly every broken load test we get pulled into at Zuplo, the numbers were already compromised before the first request went out - by the workload model (i.e. how the tool decides when to send the next request) and by the summary statistics.

So this post is about that half of the job: status-code segmentation, open workload models, percentile reporting, and k6 thresholds that enforce all three so you cannot forget them mid-test.

Segmenting latency by status code

Go back to those 30 second medians for a second, because they stop being scary the moment you know what they are. A 504 contributes exactly one value to your latency data: your timeout setting. Enough of them in a small sample and every aggregate gets dragged toward it. That is the tell to look for - error medians landing on suspiciously round numbers. 504s at almost exactly 30 seconds are your timeout configuration reporting itself back to you, not anything the API did. Blend all three populations together and the aggregate lands in the gap between them, describing none of them.

The same blending runs the other way, and that version is worse. Failing is typically the fastest thing a server does: a 503 rejected at the front of the stack returns in single-digit milliseconds while real work takes hundreds. So a system in serious trouble can look fast in aggregate, mean latency flat or improving while the error rate climbs. jeffbee makes the point in the Hacker News discussion of Marc Brooker’s post on tail latency.

The first transformation on any load test output is a group-by on status code, and every percentile you report should name its population. Which layer produced the errors is a separate question; finding where they came from takes tracing rather than better statistics.

Holding a constant arrival rate

Most load tests exist to answer a question of the form “what happens at 100 RPS?” So the test should send 100 requests every second for the whole run, whether the API is answering in 20ms or in two seconds. That sounds obvious, and it is the thing most load tests quietly fail to do.

Many tools default to a closed workload model: a fixed pool of virtual users (VUs in k6 terminology), each sending a request, waiting for the response, then sending the next. Nobody sets the arrival rate there. It is whatever falls out of the pool size divided by the response time. An open model inverts that: you set the arrival rate, requests go out on that schedule regardless of what came back, and concurrency is whatever holding the schedule turns out to require.

The difference sounds academic until the server slows down. In a closed model every slow response delays that VU’s next request, so the test backs off at exactly the moment the interesting behavior starts, thinning the dataset right where you need samples most. Real traffic typically doesn’t work that way: uncoordinated callers of a public API don’t slow down because your tail latency degraded.

Gil Tene named this measurement error coordinated omission, and his talk How Not to Measure Latency (which this post borrows its title from) is the canonical treatment. His load generator wrk2 exists to correct for it, and its README shows the size of the correction: one 1.4-second server stall made the corrected 99th percentile roughly 200x larger than the uncorrected one. The TLDR is that the miss lands on exactly the percentile most SLAs are written against.

Tene frames the failure as service time versus response time. Service time is how long the system takes to process your request once it starts (i.e. the cashier scanning your groceries). Response time also includes the wait in line, which under saturation is most of what your users experience. A closed-model test that self-throttles reports something close to service time as response time.

Even the k6 documentation on open and closed models notes that the closed variety is subject to coordinated omission. Closed models aren’t wrong everywhere - a fixed worker pool draining a queue really is a closed system - but for a public API, an open model almost always matches production.

Once you’re setting a rate rather than a concurrency, holding that rate becomes the generator’s problem rather than the API’s. The same 100 RPS against one-second responses needs five times the requests in flight it needed against 200ms responses, which means more VUs, more CPU, and past some point more machines. Size the generator for the degraded case you want to observe rather than the happy path, and check afterward that the rate really held: k6 counts the iterations it could not start on schedule in dropped_iterations, and anything above zero means you measured a slower test than the one you designed. Sizing that fleet is the next post’s subject.

Reporting percentiles

With the populations separated and the workload model open, the remaining damage happens in the reporting. I’ll assume you already report percentiles rather than means; our guide to solving latency problems in high-traffic APIs covers why averages hide tail latency. What’s left are the mistakes people make with the percentiles.

The most common is averaging them. Distributed load generators report per worker, dashboards per time window, and the instinct is to average those into one headline number. But a percentile is a rank within a population, and ranks don’t survive arithmetic across populations. Baron Schwartz’s piece on why percentiles don’t work the way you think puts it bluntly: “the math is just broken.”

Merge the underlying data instead and recompute from the combined population. HdrHistogram handles that well: its histograms merge cheaply and with bounded error. If your tooling only exposes per-worker or per-window percentiles, treat those as diagnostics and keep raw samples or mergeable histograms for anything you report.

p99 is a more mainstream experience than the name suggests. One request has a 1% chance of landing at p99 or worse, but at 100 API calls per session, a back-of-envelope calculation (1 - 0.99^100) puts the chance of at least one p99-or-worse response near 63%. That arithmetic carries a caveat I’ll get to below, but your heaviest users, the ones you least want to annoy, are the most likely to see the tail.

Tene makes one more underrated point: don’t throw the max away. It’s the worst thing your test actually observed, and the one number a percentile summary is guaranteed to hide. When it happened matters too, since a ramp-up max means something different from a steady-state one. The max and its timestamp beat another decimal place on the p95.

Building the k6 script

All of this is easy to forget mid-test, so encode the rules in the script and let the run fail on its own. Two k6 features do most of the work: arrival-rate executors give you the open model, and thresholds turn metric conditions into pass/fail criteria with a non-zero exit code, which is the part your CI pipeline reads. What follows is the shape of the script rather than a k6 tutorial; the k6 docs cover installation and every option in far more detail than a blog post should.

The scenario block below ramps to 500 RPS with the ramping-arrival-rate executor, holds there, and ramps down.

import http from "k6/http";
import { Trend } from "k6/metrics";

const BASE_URL = __ENV.TARGET_URL || "https://api.example.com";

export const options = {
  discardResponseBodies: true, // bodies we never read only cost memory here
  // so the summary prints the p(99) the thresholds below are written against
  summaryTrendStats: ["avg", "min", "med", "p(95)", "p(99)", "max"],
  scenarios: {
    steady: {
      executor: "ramping-arrival-rate",
      startRate: 50, // iterations started per timeUnit, so 50 RPS at t=0
      timeUnit: "1s",
      preAllocatedVUs: 200, // buffer over the ~100 VUs the happy path needs
      maxVUs: 500, // headroom for when responses slow down
      stages: [
        { target: 500, duration: "2m" }, // ramp to 500 RPS
        { target: 500, duration: "5m" }, // hold there
        { target: 0, duration: "1m" }, // ramp down
      ],
    },
  },
};

The executor starts iterations on schedule whether or not earlier requests have completed, which is the open model from above. The two VU numbers come from arrival rate times response time: 500 RPS at 200ms responses needs roughly 100 VUs, the same rate at one-second responses needs 500, so the pool gets sized for the degraded case rather than the happy one.

Thresholds go in that same options object, as a thresholds key alongside scenarios:

  thresholds: {
    // delayAbortEval keeps one unlucky early request from killing the run
    http_req_failed: [
      { threshold: "rate<0.01", abortOnFail: true, delayAbortEval: "30s" },
    ],
    // Aggregate guardrail, only trustworthy while failures stay rare
    http_req_duration: ["p(95)<300", "p(99)<800"],
    // The one that reflects what users actually got
    duration_2xx: ["p(95)<300", "p(99)<800"],
  },

http_req_failed matters most: past 1% failures the latency numbers are polluted in the ways covered above, so the run aborts rather than burning generator time on an unusable dataset. The duration values are placeholders, and yours should come from your service-level objective (SLO) rather than from what the test happens to produce. Thresholding http_req_duration alone would repeat the blending mistake from the top of this post, which is what the third threshold on a successes-only metric is there to avoid.

That duration_2xx metric and its error siblings are custom Trend metrics fed in the default function:

// Successes and errors are different populations, so each gets its own trend.
// The second arg marks these as times, which is why k6 prints them in ms.
const duration2xx = new Trend("duration_2xx", true);
const duration4xx = new Trend("duration_4xx", true);
const duration5xx = new Trend("duration_5xx", true);

export default function () {
  const res = http.get(`${BASE_URL}/items`);

  if (res.status >= 200 && res.status < 300) {
    // k6 tag values are strings, so convert the status code
    duration2xx.add(res.timings.duration, { status: String(res.status) });
  } else if (res.status >= 400 && res.status < 500) {
    duration4xx.add(res.timings.duration, { status: String(res.status) });
  } else {
    // 5xx, plus status 0 for requests that never got a response
    duration5xx.add(res.timings.duration, { status: String(res.status) });
  }
}

Each duration lands in the trend matching its status class, so the summary prints separate percentile lines for duration_2xx, duration_4xx, and duration_5xx with no post-processing, and the exact status code rides along as a tag so you can still pull the 503s apart from the 504s.

The three snippets assemble into one file: imports, BASE_URL, and metric declarations at the top, then the options object, then the default function. Point TARGET_URL at something that isn’t production and run it with k6 run. A badly designed test can still pass good thresholds, so this won’t catch everything, but the failure modes above surface in the summary instead of hiding in an aggregate.

Limits of the percentile math

Here’s the caveat from earlier: the 1 - 0.99^100 session math assumes every request’s latency is drawn independently from the same distribution, and in a real system it isn’t: slowness clusters around causes like one slow node, a GC pause, an expired cache, a hot tenant. Correlated tails mean fewer users hit the tail than the formula predicts, but the ones who do tend to hit it repeatedly. The direction of the argument survives, the precise percentage doesn’t, so don’t present “63% of users see p99” as a measurement of your system.

None of this makes averages useless. Averages compose (you can add, scale, and multiply them by rates), which makes them the right tool for capacity arithmetic. Little’s law (concurrency equals arrival rate times average time in system) runs on the mean, and so does most throughput and cost math, including the VU sizing above. Use averages to size systems and percentiles to describe experience; problems show up when one does the other’s job.

Percentiles also stop being meaningful at small sample sizes. In a 500-request test about five samples determine the p99, and it moves between runs for reasons unrelated to the API. For short tests, and for systems with hard deadlines (i.e. anywhere one response past N seconds is an incident), the max is typically the more honest summary. With a few hundred samples, report the median and the max, and resist printing a p99.

Check the test before you page anyone

The customer at the top of this post did not have a broken API. They had a broken summary. That is the single most useful thing to take from this: when a load test hands you an alarming number, the test is more likely to be wrong than the system, and it costs you ten minutes to rule that out before you escalate.

Three habits get you most of the way there:

• Group by status code first, every time. No percentile gets reported without naming which population it came from. That one transformation is what turned an overnight page into a non-event.
• Keep the arrival rate independent of response times. Use an open workload model so the test does not quietly back off at the exact moment the system starts misbehaving.
• Never average a percentile. Merge the raw samples and recompute, or label the numbers per worker and stop treating them as a headline.

That is the statistics half of load testing. The other half is physical: where you run your generators decides what you measure, along with the network path they share with your backend and how many of them there are. Our performance testing guide covers that side, including the rule to never run generators in the same cloud provider as your backend - a baseline that never leaves the provider’s network skews every comparison built on top of it.

If you’re about to run a serious test through a Zuplo gateway, tell us first. We’re happy to allowlist your generator IPs, watch the run from our side, and help read the results with you. We do a lot of this.