---
title: "How Not to Load Test an API"
description: "The same load test data can tell you an API is fine or failing, depending on how you summarize it. How to segment results, pick a workload model that doesn't back off when the API slows down, and report percentiles honestly, with a runnable k6 setup."
canonicalUrl: "https://zuplo.com/blog/2026/07/28/how-not-to-load-test-an-api"
pageType: "blog"
date: "2026-07-28"
authors: "nate"
tags: "API Gateway, API Best Practices"
image: "https://zuplo.com/og?text=How%20Not%20to%20Load%20Test%20an%20API"
---
A large customer woke people up over an overnight performance test. A few
thousand requests spread across about two hours, one aggregate latency number,
and that number was bad enough to page on. As far as anyone on that team could
tell, their API had fallen over in the middle of the night.
It hadn't. We took the same data and grouped it by status code, and three
completely different stories fell out of the one number. The 504s had medians
sitting right at the 30 second mark. The 503s came back in about five seconds.
The successful 200s came back around 280ms, which is what they had been doing
all night. Nobody had misconfigured anything and the API was fine. One summary
statistic had averaged timeouts, rejections, and successes together into a
number that described none of them, and that is the number that got escalated.
Generating load against an API is not the hard part of load testing; any modern
tool will happily produce thousands of requests per second from a laptop. In
nearly every broken load test we get pulled into at Zuplo, the numbers were
already compromised before the first request went out - by the workload model
(i.e. how the tool decides when to send the next request) and by the summary
statistics.
So this post is about that half of the job: status-code segmentation, open
workload models, percentile reporting, and k6 thresholds that enforce all three
so you cannot forget them mid-test.
## Segmenting latency by status code
Go back to those 30 second medians for a second, because they stop being scary
the moment you know what they are. A 504 contributes exactly one value to your
latency data: your timeout setting. Enough of them in a small sample and every
aggregate gets dragged toward it. That is the tell to look for - error medians
landing on suspiciously round numbers. 504s at almost exactly 30 seconds are
your timeout configuration reporting itself back to you, not anything the API
did. Blend all three populations together and the aggregate lands in the gap
between them, describing none of them.
The same blending runs the other way, and that version is worse. Failing is
typically the fastest thing a server does: a 503 rejected at the front of the
stack returns in single-digit milliseconds while real work takes hundreds. So a
system in serious trouble can look fast in aggregate, mean latency flat or
improving while the error rate climbs. jeffbee makes the point in the
[Hacker News discussion](https://news.ycombinator.com/item?id=26864908) of
[Marc Brooker's post on tail latency](https://brooker.co.za/blog/2021/04/19/latency.html).
The first transformation on any load test output is a group-by on status code,
and every percentile you report should name its population. Which layer produced
the errors is a separate question;
[finding where they came from](/blog/load-testing-with-tracing) takes tracing
rather than better statistics.
## Holding a constant arrival rate
Most load tests exist to answer a question of the form "what happens at 100
RPS?" So the test should send 100 requests every second for the whole run,
whether the API is answering in 20ms or in two seconds. That sounds obvious, and
it is the thing most load tests quietly fail to do.
Many tools default to a closed workload model: a fixed pool of virtual users
(VUs in k6 terminology), each sending a request, waiting for the response, then
sending the next. Nobody sets the arrival rate there. It is whatever falls out
of the pool size divided by the response time. An open model inverts that: you
set the arrival rate, requests go out on that schedule regardless of what came
back, and concurrency is whatever holding the schedule turns out to require.
| | Closed workload model | Open workload model |
| ------------------------ | ----------------------------- | ----------------------------------------------- |
| You configure | Concurrency: a fixed VU pool | Arrival rate: iterations started per second |
| What falls out of it | The arrival rate | The concurrency, up to the VU ceiling |
| k6 executors | `constant-vus`, `ramping-vus` | `constant-arrival-rate`, `ramping-arrival-rate` |
| When responses slow down | Fewer requests get sent | The schedule holds, more VUs get used |
The difference sounds academic until the server slows down. In a closed model
every slow response delays that VU's next request, so the test backs off at
exactly the moment the interesting behavior starts, thinning the dataset right
where you need samples most. Real traffic typically doesn't work that way:
uncoordinated callers of a public API don't slow down because your tail latency
degraded.
Gil Tene named this measurement error coordinated omission, and his talk
[How Not to Measure Latency](https://www.infoq.com/presentations/latency-response-time/)
(which this post borrows its title from) is the canonical treatment. His load
generator [wrk2](https://github.com/giltene/wrk2) exists to correct for it, and
its README shows the size of the correction: one 1.4-second server stall made
the corrected 99th percentile roughly 200x larger than the uncorrected one. The
TLDR is that the miss lands on exactly the percentile most SLAs are written
against.
Tene frames the failure as service time versus response time. Service time is
how long the system takes to process your request once it starts (i.e. the
cashier scanning your groceries). Response time also includes the wait in line,
which under saturation is most of what your users experience. A closed-model
test that self-throttles reports something close to service time as response
time.
Even the
[k6 documentation on open and closed models](https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/open-vs-closed/)
notes that the closed variety is subject to coordinated omission. Closed models
aren't wrong everywhere - a fixed worker pool draining a queue really is a
closed system - but for a public API, an open model almost always matches
production.
Once you're setting a rate rather than a concurrency, holding that rate becomes
the generator's problem rather than the API's. The same 100 RPS against
one-second responses needs five times the requests in flight it needed against
200ms responses, which means more VUs, more CPU, and past some point more
machines. Size the generator for the degraded case you want to observe rather
than the happy path, and check afterward that the rate really held: k6 counts
the iterations it could not start on schedule in
[dropped_iterations](https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/dropped-iterations/),
and anything above zero means you measured a slower test than the one you
designed. Sizing that fleet is
[the next post's subject](/blog/load-test-from-multiple-regions).
## Reporting percentiles
With the populations separated and the workload model open, the remaining damage
happens in the reporting. I'll assume you already report percentiles rather than
means; our guide to
[solving latency problems in high-traffic APIs](/learning-center/solving-latency-problems-in-high-traffic-apis)
covers why averages hide tail latency. What's left are the mistakes people make
with the percentiles.
The most common is averaging them. Distributed load generators report per
worker, dashboards per time window, and the instinct is to average those into
one headline number. But a percentile is a rank within a population, and ranks
don't survive arithmetic across populations. Baron Schwartz's piece on
[why percentiles don't work the way you think](https://www.solarwinds.com/blog/why-percentiles-dont-work-the-way-you-think)
puts it bluntly: "the math is just broken."
Merge the underlying data instead and recompute from the combined population.
[HdrHistogram](http://hdrhistogram.org/) handles that well: its histograms merge
cheaply and with bounded error. If your tooling only exposes per-worker or
per-window percentiles, treat those as diagnostics and keep raw samples or
mergeable histograms for anything you report.
A distributed test report shows a latency table per generator, and someone
averages the ten p99 rows into a single number for the summary slide. That
average is not the p99 of the combined traffic, and depending on how load was
distributed it can miss in either direction. Merge the raw data and recompute,
or label the numbers as per-worker.
p99 is a more mainstream experience than the name suggests. One request has a 1%
chance of landing at p99 or worse, but at 100 API calls per session, a
back-of-envelope calculation (1 - 0.99^100) puts the chance of at least one
p99-or-worse response near 63%. That arithmetic carries a caveat I'll get to
below, but your heaviest users, the ones you least want to annoy, are the most
likely to see the tail.
Tene makes one more underrated point: don't throw the max away. It's the worst
thing your test actually observed, and the one number a percentile summary is
guaranteed to hide. When it happened matters too, since a ramp-up max means
something different from a steady-state one. The max and its timestamp beat
another decimal place on the p95.
## Building the k6 script
All of this is easy to forget mid-test, so encode the rules in the script and
let the run fail on its own. Two k6 features do most of the work: arrival-rate
executors give you the open model, and
[thresholds](https://grafana.com/docs/k6/latest/using-k6/thresholds/) turn
metric conditions into pass/fail criteria with a non-zero exit code, which is
the part your CI pipeline reads. What follows is the shape of the script rather
than a k6 tutorial; the [k6 docs](https://grafana.com/docs/k6/latest/) cover
installation and every option in far more detail than a blog post should.
The scenario block below ramps to 500 RPS with the
[`ramping-arrival-rate` executor](https://grafana.com/docs/k6/latest/using-k6/scenarios/executors/ramping-arrival-rate/),
holds there, and ramps down.
```javascript
import http from "k6/http";
import { Trend } from "k6/metrics";
const BASE_URL = __ENV.TARGET_URL || "https://api.example.com";
export const options = {
discardResponseBodies: true, // bodies we never read only cost memory here
// so the summary prints the p(99) the thresholds below are written against
summaryTrendStats: ["avg", "min", "med", "p(95)", "p(99)", "max"],
scenarios: {
steady: {
executor: "ramping-arrival-rate",
startRate: 50, // iterations started per timeUnit, so 50 RPS at t=0
timeUnit: "1s",
preAllocatedVUs: 200, // buffer over the ~100 VUs the happy path needs
maxVUs: 500, // headroom for when responses slow down
stages: [
{ target: 500, duration: "2m" }, // ramp to 500 RPS
{ target: 500, duration: "5m" }, // hold there
{ target: 0, duration: "1m" }, // ramp down
],
},
},
};
```
The executor starts iterations on schedule whether or not earlier requests have
completed, which is the open model from above. The two VU numbers come from
arrival rate times response time: 500 RPS at 200ms responses needs roughly 100
VUs, the same rate at one-second responses needs 500, so the pool gets sized for
the degraded case rather than the happy one.
Thresholds go in that same `options` object, as a `thresholds` key alongside
`scenarios`:
```javascript
thresholds: {
// delayAbortEval keeps one unlucky early request from killing the run
http_req_failed: [
{ threshold: "rate<0.01", abortOnFail: true, delayAbortEval: "30s" },
],
// Aggregate guardrail, only trustworthy while failures stay rare
http_req_duration: ["p(95)<300", "p(99)<800"],
// The one that reflects what users actually got
duration_2xx: ["p(95)<300", "p(99)<800"],
},
```
`http_req_failed` matters most: past 1% failures the latency numbers are
polluted in the ways covered above, so the run aborts rather than burning
generator time on an unusable dataset. The duration values are placeholders, and
yours should come from your service-level objective (SLO) rather than from what
the test happens to produce. Thresholding `http_req_duration` alone would repeat
the blending mistake from the top of this post, which is what the third
threshold on a successes-only metric is there to avoid.
That `duration_2xx` metric and its error siblings are custom `Trend` metrics fed
in the default function:
```javascript
// Successes and errors are different populations, so each gets its own trend.
// The second arg marks these as times, which is why k6 prints them in ms.
const duration2xx = new Trend("duration_2xx", true);
const duration4xx = new Trend("duration_4xx", true);
const duration5xx = new Trend("duration_5xx", true);
export default function () {
const res = http.get(`${BASE_URL}/items`);
if (res.status >= 200 && res.status < 300) {
// k6 tag values are strings, so convert the status code
duration2xx.add(res.timings.duration, { status: String(res.status) });
} else if (res.status >= 400 && res.status < 500) {
duration4xx.add(res.timings.duration, { status: String(res.status) });
} else {
// 5xx, plus status 0 for requests that never got a response
duration5xx.add(res.timings.duration, { status: String(res.status) });
}
}
```
Each duration lands in the trend matching its status class, so the summary
prints separate percentile lines for `duration_2xx`, `duration_4xx`, and
`duration_5xx` with no post-processing, and the exact status code rides along as
a tag so you can still pull the 503s apart from the 504s.
The three snippets assemble into one file: imports, `BASE_URL`, and metric
declarations at the top, then the `options` object, then the default function.
Point `TARGET_URL` at something that isn't production and run it with `k6 run`.
A badly designed test can still pass good thresholds, so this won't catch
everything, but the failure modes above surface in the summary instead of hiding
in an aggregate.
## Limits of the percentile math
Here's the caveat from earlier: the 1 - 0.99^100 session math assumes every
request's latency is drawn independently from the same distribution, and in a
real system it isn't: slowness clusters around causes like one slow node, a GC
pause, an expired cache, a hot tenant. Correlated tails mean fewer users hit the
tail than the formula predicts, but the ones who do tend to hit it repeatedly.
The direction of the argument survives, the precise percentage doesn't, so don't
present "63% of users see p99" as a measurement of your system.
None of this makes averages useless. Averages compose (you can add, scale, and
multiply them by rates), which makes them the right tool for capacity
arithmetic. Little's law (concurrency equals arrival rate times average time in
system) runs on the mean, and so does most throughput and cost math, including
the VU sizing above. Use averages to size systems and percentiles to describe
experience; problems show up when one does the other's job.
Percentiles also stop being meaningful at small sample sizes. In a 500-request
test about five samples determine the p99, and it moves between runs for reasons
unrelated to the API. For short tests, and for systems with hard deadlines (i.e.
anywhere one response past N seconds is an incident), the max is typically the
more honest summary. With a few hundred samples, report the median and the max,
and resist printing a p99.
## Check the test before you page anyone
The customer at the top of this post did not have a broken API. They had a
broken summary. That is the single most useful thing to take from this: when a
load test hands you an alarming number, the test is more likely to be wrong than
the system, and it costs you ten minutes to rule that out before you escalate.
Three habits get you most of the way there:
- **Group by status code first, every time.** No percentile gets reported
without naming which population it came from. That one transformation is what
turned an overnight page into a non-event.
- **Keep the arrival rate independent of response times.** Use an open workload
model so the test does not quietly back off at the exact moment the system
starts misbehaving.
- **Never average a percentile.** Merge the raw samples and recompute, or label
the numbers per worker and stop treating them as a headline.
That is the statistics half of load testing. The other half is physical:
[where you run your generators decides what you measure](/blog/load-test-from-multiple-regions),
along with the network path they share with your backend and how many of them
there are. Our [performance testing guide](/docs/articles/performance-testing)
covers that side, including the rule to never run generators in the same cloud
provider as your backend - a baseline that never leaves the provider's network
skews every comparison built on top of it.
If you're about to run a serious test through a Zuplo gateway, tell us first.
We're happy to allowlist your generator IPs, watch the run from our side, and
help read the results with you. We do a lot of this.