---
title: "How Not to Load Test an API"
description: "A load test can make a failing API look fast and a healthy API look slow. Learn how to segment results, choose the right workload model, and report percentiles without distorting them."
canonicalUrl: "https://zuplo.com/blog/2026/07/28/how-not-to-load-test-an-api"
pageType: "blog"
date: "2026-07-28"
authors: "nate"
tags: "API Gateway, API Best Practices"
image: "https://zuplo.com/og?text=How%20Not%20to%20Load%20Test%20an%20API"
---
In nearly every broken load test we get pulled into at Zuplo, the first problem
is the test, not the API. Generating thousands of requests per second is easy.
Making sure the result describes what happened is the hard part.
We usually find one of three mistakes: successful and failed requests combined
into one latency distribution, a workload model that backs off when the API
slows down, or percentiles averaged across separate workers. Any one of them can
turn a clean run into a bad report.
## Separate latency by status code before reading it
A 200, a 503, and a 504 describe different work. Putting them in one latency
distribution produces a number that describes none of them. This is the first
thing we check when a report does not match the API's production telemetry.
A 504 often lands near a configured timeout, such as 30 seconds. That value
mostly tells you which deadline expired. In the run below, the 504s clustered
around 30 seconds, the 503s returned in about five seconds, and the successful
responses stayed around 280ms. One aggregate would have hidden all three
populations.
The inverse is worse. A 503 can return in a few milliseconds because rejecting
work is cheaper than completing it. If failures increase, aggregate latency can
improve while the API is getting worse.
Group results by status code or status class first. Every percentile in a report
should name the population it covers, such as "p99 for 2xx responses." If you
need to identify which layer produced an error, use
[distributed tracing](/blog/load-testing-with-tracing). Better summary
statistics cannot localize a failure.
## Use an open workload model for a fixed request rate
If you want to know what happens at 100 requests per second, the test must keep
sending 100 requests per second when the API slows down.
Many tools default to a closed workload model. A fixed pool of virtual users
sends requests in a loop, and each user waits for a response before sending the
next request. When latency rises, the tool sends fewer requests. The test backs
off at the point where the system becomes interesting.
An open workload model schedules new requests at a fixed arrival rate. Slower
responses increase concurrency, but they do not reduce the requested rate.
| | Closed workload model | Open workload model |
| ------------------------ | ----------------------------- | ----------------------------------------------- |
| You configure | Concurrency: a fixed VU pool | Arrival rate: iterations started per second |
| What the test determines | Arrival rate | Concurrency, up to the VU ceiling |
| k6 executors | `constant-vus`, `ramping-vus` | `constant-arrival-rate`, `ramping-arrival-rate` |
| When responses slow down | Fewer requests are sent | More VUs are used to hold the schedule |
Gil Tene calls the error in the closed model
[coordinated omission](https://www.infoq.com/presentations/latency-response-time/).
The generator omits samples from the period when requests would have queued.
That makes tail latency look better than what users experience. The k6
[open and closed workload model documentation](https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/open-vs-closed/)
describes the same limitation.
Closed models are valid for genuinely closed systems, such as a fixed worker
pool draining a queue. Public API traffic usually behaves like an open system:
independent clients do not coordinate to reduce traffic when your API slows
down.
An open model shifts one responsibility to the generator. It needs enough VUs,
CPU, memory, and network capacity to maintain the schedule during the degraded
case. Check k6's
[`dropped_iterations`](https://grafana.com/docs/k6/latest/using-k6/scenarios/concepts/dropped-iterations/)
after every run. Any value above zero means the generator failed to send the
workload you configured.
## Recompute percentiles from the combined population
The most tempting reporting mistake is averaging percentiles from workers or
time windows. A percentile is a rank within one population, and ranks do not
compose through arithmetic. The average of ten worker-level p99 values is not
the p99 of their combined requests, no matter how reasonable the spreadsheet
looks.
Merge the underlying samples and recompute the percentile. If retaining every
sample is too expensive, use a mergeable histogram such as
[HdrHistogram](http://hdrhistogram.org/). If your tooling exposes only
per-worker percentiles, label them as per-worker diagnostics instead of turning
them into one headline number. Baron Schwartz's explanation of
[why percentiles cannot be averaged](https://www.solarwinds.com/blog/why-percentiles-dont-work-the-way-you-think)
goes deeper into the math.
Never average the p99 rows from a distributed test. Merge the samples or
histograms, then compute p99 from the combined population.
Report the maximum too. It is the worst latency the test observed and the value
that a percentile summary always hides. Include its timestamp so readers can
tell whether it happened during warm-up, steady state, or ramp-down.
## Encode the measurement rules in k6
k6 arrival-rate executors provide the open workload model. Thresholds turn your
measurement requirements into pass or fail conditions that CI can enforce.
This scenario ramps to 500 RPS, holds for five minutes, then ramps down:
```javascript
import http from "k6/http";
import { Trend } from "k6/metrics";
const BASE_URL = __ENV.TARGET_URL || "https://api.example.com";
const duration2xx = new Trend("duration_2xx", true);
const duration4xx = new Trend("duration_4xx", true);
const duration5xx = new Trend("duration_5xx", true);
export const options = {
discardResponseBodies: true,
summaryTrendStats: ["avg", "min", "med", "p(95)", "p(99)", "max"],
scenarios: {
steady: {
executor: "ramping-arrival-rate",
startRate: 50,
timeUnit: "1s",
preAllocatedVUs: 200,
maxVUs: 500,
stages: [
{ target: 500, duration: "2m" },
{ target: 500, duration: "5m" },
{ target: 0, duration: "1m" },
],
},
},
thresholds: {
http_req_failed: [
{ threshold: "rate<0.01", abortOnFail: true, delayAbortEval: "30s" },
],
duration_2xx: ["p(95)<300", "p(99)<800"],
dropped_iterations: ["count==0"],
},
};
export default function () {
const res = http.get(`${BASE_URL}/items`);
if (res.status >= 200 && res.status < 300) {
duration2xx.add(res.timings.duration, { status: String(res.status) });
} else if (res.status >= 400 && res.status < 500) {
duration4xx.add(res.timings.duration, { status: String(res.status) });
} else {
duration5xx.add(res.timings.duration, { status: String(res.status) });
}
}
```
The VU limits should cover the latency you expect during degradation. At 500
RPS, 200ms responses require roughly 100 concurrent VUs. One-second responses
require roughly 500. Those estimates come from Little's law: concurrency equals
arrival rate multiplied by average time in the system.
The 2xx threshold prevents fast failures from making the run look healthy. The
failure-rate threshold aborts after failures exceed 1%, following a 30-second
grace period. Replace the example latency values with thresholds derived from
your SLO.
The custom trends keep 2xx, 4xx, and 5xx response times separate in the summary.
The exact status code remains available as a tag, so you can split 503s from
504s during analysis.
## Know when a percentile is the wrong summary
Percentiles become unstable with small samples. In a 500-request test, only a
handful of observations determine p99. For short runs, report the median and
maximum instead of presenting a noisy p99 as precise.
Averages still have an important job. They compose cleanly, which makes them
useful for throughput, concurrency, and cost calculations. Use averages for
capacity math and percentiles for user experience.
Also be careful when translating request percentiles into user impact. The claim
that a user making 100 requests has a 63% chance of seeing p99 latency assumes
independent samples from one stable distribution. Real latency is correlated
around slow nodes, cache misses, garbage collection, and hot tenants. Use
session-level data if you want to make a claim about sessions.
## Validate the test before blaming the API
When someone hands us an alarming load-test number, we check four things before
blaming the API:
1. Segment latency and errors by status code.
2. Confirm the arrival rate held for the full measurement window.
3. Confirm `dropped_iterations` stayed at zero and the generators had spare
capacity.
4. Recompute aggregate percentiles from combined samples or histograms.
The next step is physical placement. A correct workload model can still produce
a biased result if the generators share a cloud provider with the backend or all
hit one region. Read
[how to place load generators across regions](/blog/load-test-from-multiple-regions)
before choosing your test infrastructure.