One of the best things about Zuplo is it's programmable nature. That combined with our approach to making policies composable means you can do some amazing things with them, like our rate-limiter. In this video we show how you can have the rate-limiter interact with external services and data. Here we use supabase as a data-source for the limits.

Here's the key code from the sample

import {
  ZuploContext,
  ZuploRequest,
  ZoneCache,
  CustomRateLimitDetails,
} from "@zuplo/runtime";
import { createClient } from "@supabase/supabase-js";

const fallBackLimits = {
  requestsAllowed: 10000,
  timeWindowMinutes: 1,
};

const CACHE_NAME = "rate-limit-cache";
const CACHE_KEY = "rate-limit-data";

export async function getRateLimit(
  request: ZuploRequest,
  context: ZuploContext,
  policyName: string
) {
  const limitResponse: CustomRateLimitDetails = {
    key: request.user.sub,
    ...fallBackLimits,
  };

  const userGroup = request.user.data.rateLimitGroup;
  const cache = new ZoneCache(CACHE_NAME, context);

  const cached: any = await cache.get(CACHE_KEY);

  if (cached) {
    context.log.debug("cache hit");
    const item = cached.find((row) => row.userGroup === userGroup);
    limitResponse.requestsAllowed = item.reqPerMinute;
    return limitResponse;
  }

  context.log.debug("cache miss");
  const supabase = createClient(
    "https://YOUR_SUPABASE_URL.supabase.co",
    "YOUR_SUPABASE_TOKEN"
  );
  const { data, error } = await supabase.from("rate-limits").select();

  if (error) {
    context.log.error(`Error reading data from supabase`, error);
    // return fallback rate-limit - don't want API downtime
    // if this dependency is down.
    return limitResponse;
  }

  const item = data.find((row) => row.userGroup === userGroup);

  if (!item) {
    context.log.warn(`No row rateLimitId '${userGroup}' found, using fallback`);
    // return fallback
    return limitResponse;
  }

  void cache.put(CACHE_KEY, data, 10);

  limitResponse.requestsAllowed = item.reqPerMinute;
  return limitResponse;
}

You could make this even higher performance by having the cache have a longer expiry, but periodically reloading the data from supabase asynchronously and pushing the results back into the cache; something like an SWR (stale, while revalidate) approach.

Get started with Zuplo for free today: Sign Up Free

See also:

Shipping a public API backed by Supabase

API Authentication using Supabase JWT tokens

Many public APIs choose to use API keys as their authentication mechanism, and with good reason. In this article, we’ll discuss how to approach API key security for your API, including:

• why you should consider API key security
• design options and tradeoffs
• best practices of API key authentication
• technical details of a sound implementation

This article is language agnostic and doesn't provide a particular solution for PHP, Python, TypeScript, C# etc but every language should afford the capabilities that would allow you to build an appropriate solution.

There is an accompanying video presentation of this content: API Key Authentication Best Practices

Why API Keys? Why not?​

I talked about this in more detail in Wait, you’re not using API Keys? but in summary, API keys are a great choice because they are plenty secure, easier for developers to use vs JWT tokens, are opaque strings that don’t give away any clues to your claims structure, and are used by some of the best API-first companies in the world like Stripe, Twilio, and SendGrid.

Perhaps the most legitimate complaint against API keys is that they are not standardized, which is true, but — thanks to programs like GitHub’s secret scanning program — some patterns are starting to emerge.

If you’re building a public API, API-key authentication is much easier for a developer to configure and learn. They work great in curl and, provided you follow some of the best practices outlined here, are plenty secure.

The main case where I would not advocate for using API keys is for operations that are on behalf of an individual user. For this, OAuth and JWT is a much better fit. Examples of APIs that do and should use OAuth are Twitter and Facebook. However, if you’re Stripe and the callee of your API is an ‘organization’ and not a user, API keys are a great choice. Perhaps the best example of this is the GitHub API, which uses both: API-keys for organization-level interactions and JWT for on-behalf of users.

Decisions to make​

The best practices for API key authentication are becoming somewhat recognizable now, but there is a dimension where we still see some variability in the implementation of API keys: to make the key retrievable or irretrievable.

The world of API-key implementations is divided into two groups. The first will show you your API key only once. You'll need to copy it and save it somewhere safe before leaving the console. This is irretrievable. Typically the keys are unrecoverable because they are not actually stored in the key database, only a hash of the key is stored in the database. This means, if lost, the keys can genuinely never be recovered. Of course, in the case of a loss you can usually regenerate a new key and view it once.

The other group allows you to go back to your developer portal and retrieve your key at any time. These keys are typically stored encrypted in the database. Meaning if the database is stolen, the thief would also need the encryption codes to access the API keys.

The tradeoffs here are tricky, and there are two schools of thought

1. Irretrievable is better because it’s more secure. The keys are stored via a one-way encryption process so they can never be retrieved, or stolen from the database in a worse case scenario.
2. Retrievable offers good-enough security with some advantages, and it’s easier to use. The keys are stored encrypted via reversible encryption. One potential security advantage is that users are less likely to feel pressured to quickly store the key somewhere to avoid losing it. A person that follows best practices will use a vault or service like 1password. However, some users will take the convenient path and paste it into a .txt file for a few minutes thinking, “I’ll delete that later.”

So what are some examples of APIs that support recoverable vs. unrecoverable today??

Irretrievable: Stripe, Amazon AWS

Retrievable: Twilio, AirTable, RapidAPI

There is some correlation between services that protect sensitive information and services seem more likely to use irretrievable, while services that are less sensitive choose retrievable for ease of use and good-enough security.

The choice is yours. Personally, I lean a little toward retrievable because I know that I personally have made the mistake of quickly pasting a newly generated irretrievable key into notepad and forgetting about it. You may come to a different conclusion for your own API key authentication.

Best Practices of API Key Authentication​

The bit of API key authentication advice you’ve been waiting for… the best practices of API key auth based on the patterns observed in the API world and our experience building our own API key authentication service for Zuplo.

1/ Secure storage for the keys

Depending on your choice of retrievable vs. irretrievable, you’ll need to take a different path. For irretrievable keys, it’s best to store them as a hash, probably using sha256 and ensure that they are stored as primary key (this will help avoid hash collisions, unlikely but possible so you should build in a retry on create and insert).

For retrievable, you’ll need to use encryption so that the values can be read from the database to show to the user at a later date. You have a few choices here, like storing the keys in a secure vault, or using encryption programmatically to store in a standard database and manage the keys yourself.

2/ Support a rolling transition period

It’s critical that you allow your users to roll their API keys in case they accidentally expose it, or just have a practice of periodically changing them out. It’s important that this ‘roll’ function either allows for multiple keys to exist at the same time or allows the setting of an expiry period on the previous key, otherwise rolling the key will cause downtime for any consumers that didn’t get the chance to plug-in the new key before the last one expired. Here’s Stripe’s roll dialog:

In Zuplo, we allow folks to have multiple keys so they can temporarily add another key and delete the old one as soon as they’re done with the transition.

3/ Show the key creation date

It’s important to show developers when the key was created so they can compare the date to any potential incidents. This is especially important if you support multiple keys so that users can differentiate between old and new.

4/ Checksum validation

Since checking the API key will be on the critical path of every API call, you want to minimize latency. This is one of the reasons that you’ll want to add a checksum to your API key. Here’s an example API key from Zuplo:

zpka_a5c5e56x54c4437fbd6ce7dee9185e_631238

The last section _631238 is a checksum that we can use to verify in the request pipeline whether this even looks like a valid key. If not, we can simply reject the request and avoid putting load on the API key store.

5/ Support secret scanning

One of the reason we have the unusual beginning of the key "zpka_" is so that we could participate in programs like GitHub’s secret scanning. This allows us to create a regular expression that allows GitHub to inform us if an API key is accidentally checked into a repo. Then we can automatically revoke the key and inform its owner of the event. We also use the checksum to double-check that it’s one of our keys before locating the service and owner.

At Zuplo, we participate in the GitHub secret scanning program so that we can offer these services to any customer using our API-key policy.

(Aside: Yes, the example key above triggered our secret scanning when I checked this article into GitHub and we got notified about a token leak 👏😆)

6/ Minimize latency and load on your API Key storage

To reduce the latency on every API request, you might consider using an in-memory cache to store keys (and any metadata read from the API Key Store). If you have a globally distributed system, you’ll want multiple caches, one in each location. Since Zuplo runs at the edge, we use a high-performance cache in every data center. To increase security it's important to consider only caching the one-way hashed version of the API-key (taking care to avoid hash-collisions by doing a pre-hash collision check at the point of key-creation, using the same hash algorithm).

You’ll need to choose an appropriate TTL (time-to-live) for your cache entries which has some tradeoffs. The longer the cache, the faster your average response time will be and less load will be placed on your API Key store - however, it will also take longer for any revocations or changes to key metadata to work.

We recommend just a couple of minutes maximum; that’s usually plenty to keep your latency low, a manageable load on your API Key Store, and be able to revoke keys quickly.

If this is important to you, you might want to design a way to actively flush a key from your cache.

7/ Hide keys until needed

Today, everybody has a high-quality camera in their pocket. Don’t show an API key on-screen unless explicitly requested. Avoid the need to show keys at all by providing a copy button. Here’s the supabase console, which almost gets full marks, but would be even better if it provided a copy option without needing me to reveal the key visually.

8/ Attention to detail — keys need to be copied and pasted

Sometimes it’s the little things in life; for example, try double-clicking on key-1 below to select it. Then try key-2.

key-1: zpka-83fff45-1639-4e8d-be-122621fcd4d1

key-2: zpka_a5c5e56x54c4437fbd6ce7dee9185e_631238

Note how much easier it is to select the API key in snake_case?

9/ Consider labeling your keys

A number of services are increasingly doing this to help their customers, but mostly to help their support teams. For example, in Stripe they have a convention as follows:

• sk_live_ - Secret Key, Live version
• pk_test_ - Publishable Key, Test version

This not only supports GitHub secret key scanning above, but it can also be invaluable to your support team when they can easily check if the customer is using the right key in the right place. Your SDK can even enforce this to prevent confusion.

The downside to key labeling is that if the key is found without context by a malicious user - they can discover which services to attack. This is one advantage of using a managed API key service that dissociates the key from any specific API. GitHub have a great article - Behind GitHub’s new authentication token formats.

A canonical flow through an API key check​

Stacking all of that together, here’s a flow chart showing the canonical implementation of the API key check using all these practices above.

Conclusion​

API keys are a great approach if you want to maximize the developer experience of those using your API, but there are quite a few things to think about when it comes to API key security. An alternative to building this yourself is to use an API Management product with a gateway that does all the work for you and includes a self-serve developer portal. Examples include Apigee, Kong, and — of course — Zuplo.

The Author​

Before founding Zuplo, Josh led Product for Stripe’s Payment Methods team (responsible for the majority of Stripes payment APIs) and worked at Facebook and Microsoft, where he founded a number of services, including Azure API Management.

Updated December 4 2022 to update recommended hashing algo to sha256 based on community feedback.

Supabase is an incredible open-source alternative to Firebase and other BaaS (Backend-as-a-service) options. The design is somewhat optimized for consumption by first-party clients like your own website or mobile app. But what if you wanted to take all that supa-ness and make an API-first product — that is a developer-friendly public API?

There is an accompanying video for this post: https://www.youtube.com/watch?v=GJSkbxMnWxE

This is where Zuplo can help. Zuplo is the fastest way to get your API to market and get a Stripe-quality experience with the three critical pillars of any API program:

• authentication
• protection (rate-limiting, firewall)
• documentation

In this example I'm going to work with a simple table that allows people to read and write entries to a supabase table that contains some reviews of skis (yes, I love to ski). Because this is an API for developers, they may be calling it from some backend service and can't login using the standard supabase method. This is where API keys are a much better choice - see Wait, you're not using API keys?.

We'll allow people, with a valid API key, to read data from the ski results table and to create new records. Hopefully it's obvious that there are many ways you can extend this example to add more behavior like roles based access, with custom policies, custom handlers and more. Come join us on Discord if you have questions or need inspiration.

Setting up Supabase​

If you haven't already, create a new project in supabase and create a table called ski-reviews with the following columns (feel free to use another domain and invent your own example):

• id (int8)
• created_at (timestamptz)
• make (varchar)
• model (varchar)
• year (int8)
• rating (int2)
• author (varchar)

Manually enter a couple of rows of data - so we have something to read from the DB.

The read all reviews route in Zuplo​

Create a new project in Zuplo - I went with supabase-ski-reviews.

Select the File tab and choose Routes. Add your first route with the following settings:

• method: GET
• path: /reviews
• summary: Get all reviews
• version: v1
• CORS: Anything goes

And in the request handler section, paste the READ ALL ROWS URL of your supabase backend (you can get to this in the **API docs section of Supabase)

• URL Rewrite: https://YOUR_SUPABASE_URL.supabase.co/rest/v1/ski-reviews?select=*
• Forward Search: unchecked

In order to call the supabase backend I need to add some authentication headers to the request before we call supabase.

Expand the Policies section of your route. Click Add policy on the Request pipeline.

First, we don't want to forward any old headers that the client sends us to Supabase so find the Clear Headers Policy and add that to your inbound pipeline. Note, that we will allow the content-type header to flow through, so this should be your policy config.

{
  "export": "ClearHeadersInboundPolicy",
  "module": "$import(@zuplo/runtime)",
  "options": {
    "exclude": ["content-type"]
  }
}

Next, we need to add the credentials to the outgoing request. We'll need to get the JWT token from supabase - you'll find it in Settings > API as shown below:

Once you've got your service_role JWT, click Add Policy again on the Request pipeline and choose the Add/Set Headers Policy and configure it as follows:

{
  "export": "SetHeadersInboundPolicy",
  "module": "$import(@zuplo/runtime)",
  "options": {
    "headers": [
      {
        "name": "apikey",
        "value": "$env(SUPABASE_API_KEY)",
        "overwrite": true
      },
      {
        "name": "authorization",
        "value": "$env(SUPABASE_AUTHZ_HEADER)",
        "overwrite": true
      }
    ]
  }
}

Save your changes.

Next, create two secret environment variables as follows:

• SUPABASE_API_KEY: "YOUR_SUPABASE_SECRET_ROLE_JWT"
• SUPABASE_AUTHZ_HEADER: "Bearer YOUR_SUPABASE_SECRET_ROLE_JWT"

Obviously, in both instances replace YOUR_SUPABASE_SECRET_ROLE_JWT with your service_role JWT from supabase.

You are now ready to invoke your API gateway and see data flow through from Supabase!

Click on the open in browser button shown below and you should see the JSON, flowing from supabase in your browser 👏.

Adding authentication​

At this point, that route is wide open to the world so we need to secure it. We'll do this using API keys. You can follow this guide Add API key Authentication. Be sure to drag the API Key authentication policy to the very top of your Request pipeline. Come back here when you're done.

Welcome back! You've now learned how to secure your API with API-Keys.

Adding a Create route​

Next we'll add a route that allows somebody to create a review. Add another route with the following settings

• method: POST
• path: /reviews
• summary: Create a new review
• version: v1
• CORS: Anything goes

And the request handler as follows:

• URL Rewrite: https://YOUR_SUPABASE_URL.supabase.co/rest/v1/ski-reviews
• Forward Search: unchecked

Expand the policies section and add the same policies (note you can reuse policies by picking from the existing policies at the top of the library)

• api-key-inbound
• clear-headers-inbound
• set-headers-inbound

Now your create route is secured, will automatically set the right headers before calling supabase. That was easy.

You can test this out by using the API Test Console to invoke your new endpoint. Go to the API Test Console and create a new test called create-review.json.

• Method: POST
• Path: /v1/reviews
• Headers:
  • content-type: application/json
  • authorization: Bearer YOUR_ZUPLO_API_KEY
• Body:

{
  "make": "Rossignol",
  "model": "Soul HD7",
  "rating": 5,
  "year": 2019
}

If you invoke your API by clicking Test you should see that you get a 201 Created - congratulations!

Add validation to your post​

To make your API more usable and more secure it is good practice to validate incoming requests. In this case we will add a JSON Schema document and use it to validate the incoming body to our POST.

Create a new schema document called new-review.json.

This example fits the ski-reviews table we described above

{
  "$id": "http://example.com/example.json",
  "type": "object",
  "default": {},
  "title": "Root Schema",
  "required": ["make", "model", "rating", "year"],
  "additionalProperties": false,
  "properties": {
    "make": {
      "type": "string",
      "default": "",
      "title": "The make Schema",
      "examples": ["DPS"]
    },
    "model": {
      "type": "string",
      "default": "",
      "title": "The model Schema",
      "examples": ["Pagoda"]
    },
    "rating": {
      "type": "integer",
      "default": 0,
      "title": "The rating Schema",
      "examples": [5]
    },
    "year": {
      "type": "integer",
      "default": 0,
      "title": "The year Schema",
      "examples": [2018]
    }
  },
  "examples": [
    {
      "make": "DPS",
      "model": "Pagoda",
      "rating": 5,
      "year": 2018,
      "author": "Josh"
    }
  ]
}

Now add a new policy to request pipeline for your Create new review route. Choose the JSON Body Validation policy and configure it to use your newly created JSON schema document:

{
  "export": "ValidateJsonSchemaInbound",
  "module": "$import(@zuplo/runtime)",
  "options": {
    "validator": "$import(./schemas/new-review.json)"
  }
}

This policy can be dragged to the first position in your pipeline.

Now to test this is working, go back to your API test console and change the body of your create-review.json test to be invalid (add a new property for example). You should find that you get a 400 Bad Request response.

Finally, lean back and marvel at your beautiful Developer Portal that took almost zero effort to get this far, wow! Hopefully you already found the link for this when adding API key support :)

Get started with Zuplo for free today: Sign Up Free

See also:

API Authentication with Supabase JWT Tokens

Supa-dynamic rate-limiting based on data (using supabase)

This guide shows you how you can use a Zuplo API gateway to add Supabase JWT Authentication and Authorization to any API, running on any cloud.

There is an accompanying video talk for this blog post: https://youtu.be/UEeSZkV7o_Y

Zuplo is an edge-based programmable API gateway that can augment any existing backend HTTP API to add JWT authentication, dynamic rate-limiting, custom transformations and projections of JSON data.

Zuplo is a fully managed service, sign-up at zuplo.com to try it out.

In this example, we’re going to add Supabase JWT authentication and authorization to a demo API and turn on rate-limiting. We’ll use the JSONPlaceholder todo API available at https://jsonplaceholder.typicode.com/todos. For ease of demonstration, this is a public API, but Zuplo has a number of options for secure connectivity to a backend API.

Step 1: Proxy your API using Zuplo​

Sign-in at portal.zuplo.com and create a new project. Click on the Routes item in the file list. You will see that you have no routes.

Click Add Route and configure the route as follows:

• Method: GET
• Path: /todos
• Summary: Get my todos
• Version: none
• CORS: Anything Goes (you can configure custom CORS policies later).

In the Request Handler section, set the URL Rewrite path to https://jsonplaceholder.typicode.com/todos

Click the save icon next to the Routes link (or press CMD+S/CTRL+S) to save your changes.

Your gateway is now ready to proxy requests to the todo API! To try it, click the Open Route link (shown below)

Step 2: Add JWT Authentication​

Next, expand the Policies section of your route and click Add Policy to the Request pipeline.

Find the [Supabase JWT Auth policy](https://zuplo.com/docs/policies/supabase-jwt-auth-inbound) and select it. Edit the Configuration to remove the requiredClaims property (we’ll set those up later) and click OK.

Finally, go to the Settings tab, choose Environment Variables, and add a new variable called SUPABASE_JWT_SECRET.

The value of the secret can be obtained from your Supabase Settings in the API section.

Save your changes by navigating back to the file tab and clicking the Save icon next to Routes. Congratulations, you have now secured your todos API with a Supabase JWT token!

You can try calling this using a JWT token from a client (web, mobile, postman, curl etc) and sending the JWT token as the Authorization header, with a value "Bearer JWT_TOKEN_HERE".

Reminder, you can get the URL of your API using the Open In Browser button we used above or, on the Getting Started page, you’ll see the root API.

Step 3 - Enforce required claims​

You can add custom claims to your user(s) in Supabase by adding them to the auth.users tables raw_app_metadata column (there’s a good article on this here).

These custom claims are encoded into the JWT token and we can use them to restrict access to our API. For this example, we updated the raw_app_metadata to have a custom claim of user_type. Note that we left the other claims in place.

UPDATE auth.users SET raw_app_meta_data =
'{"provider":"email", "providers":["email"], "user_type": "supa_user"}'
WHERE id = 'user_id_here';

Now we can require that anybody calling our API has a specific claim. To do this we update the requiredClaims property on our policy configuration. Go back to the Route Designer and find your Supabase JWT policy. Click the edit button and change the Configuration as follows:

{
  "export": "SupabaseJwtInboundPolicy",
  "module": "$import(@zuplo/runtime)",
  "options": {
    "secret": "$env(SUPABASE_JWT_SECRET)",
    "allowUnauthenticatedRequests": false,
    "requiredClaims": {
      "user_type": ["supa_users"]
    }
  }
}

This means anybody calling this particularly route must have a user_type claim of supa_user to successfully invoke this API.

Get started with Zuplo for free today: Sign Up Free

See also:

Shipping a public API backed by Supabase

Supa-dynamic rate-limiting based on data (using supabase)

Auth0 is still one of the best ways to add authorization to your app. However, one minor annoyance I have found is that there is no way to force every login to use a single identity provider (i.e. connection) without configuring each client with the connection parameter. So even for an app that only allows users to login with a single social connection (i.e. Google) users will still see the Auth0 login picker by default.

With Cloudflare Workers and an Auth0 custom domain it is easy to fix this issue. After you setup your custom domain, you need to make sure you are proxying the CNAME through Cloudflare.

Next, create a simple Cloudflare Worker with the following code.

export default {
  async fetch(request, env) {
    return await handleRequest(request);
  },
};

const AUTH0_CONNECTION = "my-connection";

async function handleRequest(request) {
  const url = new URL(request.url);
  // Checking path just in case, but this
  // worker should only run on this path
  if (url.pathname === "/authorize") {
    if (url.searchParams.get("connection")) {
      return fetch(request);
    } else {
      url.searchParams.set("connection", AUTH0_CONNECTION);
      const newRequest = new Request(url.toString(), new Request(request));
      return fetch(newRequest);
    }
  }
  return fetch(request);
}

Finally, configure a Route for the Cloudflare worker to run on the /authorize path. The route should look like this: my-cname.example.com/authorize*. Make sure to put the * at the end; otherwise, requests with the query parameters will not be sent to the worker.

When a request comes to the regular /authorize URL to start an OAuth flow, the connection query parameter is added automatically. Every user will skip the Auth0 login page and immediately go to the login page of the specified connection.

One word of caution, I am unsure if Auth0 supports this configuration. It works for me, but Auth0 could make changes that break this at some point.

Hopefully, this helps solve a minor annoyance you might have with Auth0.

Today we’re announcing that Zuplo offers API Key Scanning on GitHub for API keys generated in Zuplo.

According to the most recent GitGuardian report, in 2021 over 6 million secrets were leaked, which was 2x 2020’s total and 3 in every 1,000 commits exposed at least one secret. The massive Heroku security incident in April 2022 was caused by API Keys checked into source control. It’s no surprise then that since we opened Zuplo up publicly we’ve seen a lot of excitement about our API Key Management capabilities. We’ve written why we think API Keys are the best way to secure your API here and now we make it effortless to secure both you and your customers with API Key Scanning.

"Heroku determined that the unidentified threat actor gained access to the machine account from an archived private GitHub repository containing Heroku source code."

Respecting the developer workflow is one of our central tenets at Zuplo, which is why we designed it with GitOps in mind. Starting today, if one of the API keys for one of your APIs in Zuplo shows up in a public repo on GitHub you’ll receive an alert from Zuplo notifying you of the token and the URL where the match was found. You can also choose to have Zuplo notify your customer on your behalf.

Zuplo API Key management includes:

• secure storage and management of keys and metadata - with an admin UI and API to manage consumers.
• integrated developer portal with self-serve key management for your customers.

If you've already built your own API Key solution we can easily integrate Zuplo authentication with custom policies or even help you API key to Zuplo for even greater protection. It's never too late to make hosting your API much easier.

API Key Leak Prevention is part of our Business and Enterprise subscriptions.

UPDATE 11/28/2022 - six months after publishing this post GitHub made this announcement: To infinity and beyond: enabling the future of GitHub’s REST API with API versioning.

"If you have an integration with the REST API, you should update it now to start sending the X-GitHub-Api-Version header."

No version 👏 No service. 👏

When providing recommendations we like to use examples of great companies, the decisions they made — that often go against the grain, and why they made those decisions. One of my favorite examples of this is the fact that the best API companies tend to use API keys.

But what about great companies getting it wrong?

We recently wrote recommendations for versioning your API and had one primary piece of advice - insist that the client includes the desired version on every request.

It’s simple: No Version? No Service.

When giving examples of great APIs, I have a few ‘go-tos’:

You’ll notice one anomaly here though - the GitHub API doesn’t use URL-based versioning and doesn’t require a version. Let’s quote their docs

When using the REST API, we encourage you to request v3 via the Accept header

Note the use of the word ‘encourage’, there’s more

*Important: The default version of the API may change in the future. If you're building an application and care about the stability of the API, be sure to request a specific version in the Acceptheader as shown in the examples below.*

Source: https://docs.github.com/en/rest/overview/media-types#request-specific-version

This approach is asking for trouble and limits your options. It’s hard to run multiple versions of your API simultaneously (without defaulting to the old one 🤮) and it’s hard to know which customers are trying to use which version of your API. Maybe you can tell because of all the support calls and errors they’re getting?

That’s why we tend to recommend URL-based versioning - it’s implicit in the address of the URL what version the client is coded to consume and it can’t be skipped, lest you’ll get a 404. It’s good enough for Stripe, AirTable, and the Twilio family so it’s probably good for you.

If you do decide to go the headers route, be sure to send back a 400 if you don’t see an explicit version requested. Your error message could be “No version, no service” - that one’s on us.

We recently discussed some best practices for versioning your API with (spoiler) a strong recommendation that you should require that clients indicate the version of the API they were designed for. Go read the article for more details.

I’ve been giving a talk at a few conferences recently about how the world’s fastest-growing companies develop their public API, which included a piece on versioning. One question I commonly get from attendees after the talk is, “How do I move clients off the old version of my API?”, where clients might be internal depts, long-term customers or even teammates who don’t want to change that UI code that is dependent on v1.

In most cases, you can’t just turn off the old version. In the cases above you’ll cause business harm by breaking the marketing department or taking down one of your loyal customers. These folks have priorities and updating to /v2/ of your API is probably not at the top of the list. So how might you create more urgency?

Creating Urgency​

We do not recommend just turning off the API permanently. Instead, you can take the approach of scheduling a ‘brownout’ or timed, temporary downtime. This is where you take the API down during a low-impact period for a short amount of time, maybe at 3 am in the morning, for 2 minutes. This is probably enough to trigger a bunch of alarms and service alerts that make the impact of the upcoming breaking change clear to the consumer.

We’d recommend sharing the schedule of the planned outages so people aren’t surprised at all and know what the glideslope to actual deprecation looks like. Some businesses share that the version of the API they are using will be fully deprecated on a specific date but, knowing that some clients will not upgrade in time, activate the first brownout at this time and notify clients that they have one more week to complete their upgrade. This can you make you appear like you’re being generous, you’re giving them more time than they were told they would have, but they still get the shock of alerts firing.

Another technique to encourage folks to move off an old API, and combines well with brownouts, is to start to add deliberate latency to the old version of the API. You can use an API gateway to do this and, as time progresses, you can increase the latency to multiple seconds even - depending on the use case.

Again, this is a business decision but once you’ve decided you need to create urgency amongst consumers of your soon-to-be-deprecated, old version of your API. This is a better approach to just going dark on active customers on the scheduled date.

We introduced a sleep and brownout policy to Zuplo to make this even easier. If you want to try it out for yourself and schedule a brownout, go sign up for Zuplo.

Weekly Zoom Chat

Are you starting work on, or in the middle of building, a public customer or partner API? Register for our "Building a Customer API" weekly chat on Zoom every Thursday at 3pm ET/12pm PT and learn from featured guests and other developers building public APIs.

Planning to build a new public customer or partner API but not sure where to start your research? Connect with fellow developers and nail development of your new API by joining our meetup on Zoom, "Building a Customer API".

On June 9, 2022 at 3pm ET/12pm PT we'll be joined by Utsav Shah of fast growing startup Vanta (and host of the Software at Scale podcast). He'll share his experience building a customer API and challenges when implementing rate limiting. Then we'll have an open discussion on implementing rate limiting and exposing APIs to customers and partners.

Register Here

About Utsav Shah

Utsav Shah is a software engineer at Vanta and host of the Software at Scale podcast. Before joining Vanta, Utsav was responsible for enabling product velocity and ensuring the reliability of Dropbox’s monolith Python web application and large async systems like Cape.

About Vanta

Our mission at Vanta is to be a layer of trust on top of cloud services, to secure the internet, increase trust in software companies, and keep consumer data safe. Think of us as your automated security and compliance expert.

Building a Customer API​

At 3pm ET/12pm PT every Thursday, Zuplo hosts a 1-hour round-table discussion virtually (Zoom) to help developers plan for and build their public customer or partner APIs. We feature one attendee each week who has recently built a public customer or partner API and who will share how they approached the process and what they learned, then stick around to answer questions and discuss. This chat is limited to 25 spaces per week to keep the conversation flowing and fruitful. We'll start with a Q&A with the featured guest and then have an open discussion immediately following.

Have more questions, check out our event FAQ.

At some point, you’re going to make an update to your API that would break existing clients if they don’t change their code. That’s OK, change happens.

However, it is critical that you give yourself the option to do two things when this situation arises:

1. Support multiple versions of your API simultaneously (so that you can give older clients the opportunity to migrate to your latest version).
2. Inform a client that the version they have coded for is no longer supported

Life is much easier if you think about versioning from the beginning of the lifecycle of your API. A key decision to make is how you want to design versioning into your API; that is, how should the client communicate the version of the API they are coded to work with?

There are two primary options:

1. URL-based versioning - where the version is encoded directly in the URL, e.g. /v1/charges. This is the most common approach and is used by most large API-first companies like Stripe, Twilio, SendGrid and Airtable.
2. Header-based versioning - where the version is in a header; either a customer header like api-version: v3 or as part of the accept header, e.g. accept: application/vnd.example+json;version=1.0.

Our recommendations​

1/ Keep those options open

First and foremost, we strongly recommend that you make the version a mandatory part of all requests received by your API. So any request that doesn’t include the version should receive a 4xx error code (400 if it’s a required header, 404 if it is missing from the URL).

This is the most important decision because it means you always have both options outlined at the opening of this post.

2/ Keep it simple

After this, we recommend URL-based versioning. There is plenty of precedent in the market and it’s the easiest for developers to use - it’s easier to test with CURL, call with fetch, test in a browser if you support GET requests. It’s just easier.

3/ Use headers if you’re passionate about building a pure REST implementation

The primary reason to use headers over URL-based versioning is to avoid violating a REST principle that states a URI should refer to a unique resource (v1/foo and v2/foo could theoretically point to the same resource). However, such pure implementations of REST APIs have not proven popular and are trickier for developers to use.

4/ Don’t break rule 1

There are examples of APIs in the public domain that have a default version if the client doesn’t specify a version. Here’s GitHub’s documentation on their API:

Even though it encourages developers to use the version header, the API still works without it and just assumes v3. We think this is a mistake; if GitHub upgrades to v4 and that becomes the new default, all of those old clients that didn’t follow the best practice will experience unpredictable behavior and strange errors.