Implementing Data Compression in REST APIs with gzip and Brotli

Want faster APIs? Start with compression. gzip and Brotli are two powerful algorithms that shrink API payloads, speeding up data transfer and reducing bandwidth. Here's what you need to know:

• Why compress? Smaller payloads mean faster responses and lower costs.
• gzip vs Brotli: gzip is faster and widely compatible; Brotli offers better compression but demands more resources.
• How it works: Clients request compression via Accept-Encoding, and servers respond with Content-Encoding.
• Implementation: Use libraries or server configurations to enable gzip and Brotli. Examples include Flask, Gin, and Express setups.

Quick tip: Compress text-based formats like JSON, but skip already-compressed files like images. Balancing compression levels and performance is key. Keep reading to learn how to set it up.

How Data Compression Works in REST APIs#

In REST APIs, data compression relies on HTTP headers to manage communication between the client and server. These headers determine which compression methods are supported and which one will be applied during each request-response cycle.

HTTP Headers for Compression#

Two key HTTP headers handle data compression in REST APIs:

• Accept-Encoding: This header is sent by the client to specify the compression algorithms it supports. Common options include gzip, compress, and br (Brotli). For instance, a client might send Accept-Encoding: gzip,compress to indicate it supports both gzip and compress formats.
• Content-Encoding: The server uses this header to inform the client about the compression method applied to the response. For example, if the server compresses the data using gzip, it will respond with Content-Encoding: gzip.

To enable compression, developers building custom API clients must explicitly configure these headers.

Client-Server Negotiation Process#

The negotiation process determines which compression method to use based on the headers. When a client sends a request with an Accept-Encoding header listing its supported formats, the server reviews the options and selects one that matches.

Let's use the Rick and Morty API as an example. By sending a GET request to https://rickandmortyapi.com/api/location/20 with the Accept-Encoding: gzip header, the API returned compressed data. The server's response included Content-Encoding: gzip, confirming gzip compression was applied to the JSON payload.

If the server cannot provide a response in one of the formats listed in the Accept-Encoding header, it should return a 406 (Not Acceptable) status code. Similarly, if a client sends data in a format the server doesn’t support, the server responds with a 415 (Unsupported Media Type) status code.

When multiple compression algorithms are applied, the server lists them in the Content-Encoding header in the order they were used. This ensures the client can decompress the data correctly.

Client Compatibility Requirements#

Once the negotiation process is established, proper client configuration becomes essential. Clients need to send Accept-Encoding headers to signal which compression formats they support. Without this header, servers often default to sending uncompressed data to avoid compatibility issues.

Clients must also be equipped to decompress responses. Decompression requires additional CPU and memory, which can introduce latency. This trade-off is especially important for mobile apps or devices with limited processing power. In some cases, compressing small payloads can result in larger data sizes due to compression overhead, making it counterproductive.

To handle compressed responses effectively, clients must include decompression libraries for the algorithms they advertise in their Accept-Encoding headers.

Additionally, the Vary: Accept-Encoding header plays a critical role in caching. It tells intermediate caches to store separate versions of a resource based on the client’s compression capabilities. This ensures that each client receives the correct version - compressed or uncompressed - depending on its request.

gzip vs Brotli Comparison#

When it comes to web compression, understanding the differences between gzip and Brotli can help you make the right choice for your specific needs. While both algorithms aim to reduce file sizes and improve performance, they each bring unique strengths to the table.

gzip vs Brotli Technical Overview#

gzip has been a trusted compression tool since the 1990s. It uses the DEFLATE algorithm, which combines LZ77 and Huffman coding, to deliver fast and reliable compression. Its low CPU overhead makes it ideal for high-throughput scenarios where speed is critical.

Brotli, introduced by Google in 2013, is a more modern solution. It also uses LZ77 and Huffman coding but adds a pre-defined dictionary tailored to common web content patterns. This dictionary allows Brotli to achieve better compression ratios, especially for text-heavy data. However, this efficiency comes with higher CPU and memory demands during both compression and decompression.

The key distinction lies in their focus: gzip prioritizes speed and compatibility, while Brotli leans toward maximizing compression efficiency. This makes Brotli particularly advantageous for APIs that handle large, text-rich responses.

gzip vs Brotli Comparison Table#

How to Choose the Right Compression Algorithm#

The decision between gzip and Brotli depends on your API's needs. If wide compatibility and low resource usage are priorities - such as in high-traffic environments or when dealing with small payloads - gzip is the way to go. Its speed and reliability make it a solid choice for legacy systems and resource-constrained setups.

On the other hand, Brotli is ideal for APIs delivering large, data-heavy responses. Whether you're serving detailed catalogs, complex JSON structures, or other content-rich data, Brotli's superior compression ratio can save significant bandwidth. This is especially beneficial for web and mobile clients with limited data plans.

For the best of both worlds, consider supporting both algorithms. Use the client's Accept-Encoding header to determine the preferred compression method. Configure your server to serve Brotli when supported and fall back to gzip for older or less capable clients.

Lastly, ensure your infrastructure - including CDNs and reverse proxies - supports the chosen compression methods. Some older caching systems may not handle Brotli properly, so it's essential to test compatibility before deployment. Next, we’ll explore how to implement these compression techniques in your REST API.

Over 10,000 developers trust Zuplo to secure, document, and monetize their APIs

Learn More

How to Implement gzip and Brotli in REST APIs#

Adding compression to REST APIs can make data transfer faster and more efficient. Here's how to set it up across different technologies.

Prerequisites for API Compression#

Before jumping into the code, ensure your setup supports the required compression methods. Most modern web servers and frameworks come with gzip built-in, and Brotli is now commonly supported in newer versions.

Focus on compressing text-based formats like JSON, XML, HTML, CSS, and JavaScript, as they compress well. Avoid compressing binary formats like images, videos, or already-compressed files, as this can actually increase their size.

Also, always use HTTPS in production to avoid potential vulnerabilities, and confirm that clients can handle the compression methods you implement.

Code Examples by Programming Language#

Here’s how to enable compression in popular programming languages:

Python with Flask#

Flask makes it simple to enable compression using the Flask-Compress library. Install it with:

pip install Flask-Compress

Then configure it in your app:

from flask import Flask, jsonify
from flask_compress import Compress

app = Flask(__name__)
Compress(app)

# Define MIME types and compression levels
app.config['COMPRESS_MIMETYPES'] = [
    'text/html', 'text/css', 'text/xml',
    'application/json', 'application/javascript'
]
app.config['COMPRESS_LEVEL'] = 6
app.config['COMPRESS_BR_LEVEL'] = 4

@app.route('/api/data')
def get_data():
    data = {"users": [{"id": i, "name": f"User {i}"} for i in range(1000)]}
    return jsonify(data)

Go with Gin Framework#

In Go, you can add compression using the Gin framework and its gzip middleware:

package main

import (
    "net/http"
    "github.com/gin-gonic/gin"
    "github.com/gin-contrib/gzip"
)

func main() {
    r := gin.Default()

    // Enable gzip compression
    r.Use(gzip.Gzip(gzip.DefaultCompression))

    r.GET("/api/data", func(c *gin.Context) {
        data := map[string]interface{}{
            "message": "This response will be compressed",
            "items": make([]int, 1000),
        }
        c.JSON(http.StatusOK, data)
    })

    r.Run(":8080")
}

Node.js with Express#

For Node.js, the compression middleware makes it easy to enable gzip:

const express = require("express");
const compression = require("compression");

const app = express();

// Configure gzip compression
app.use(
  compression({
    level: 6, // Compression level (1-9)
    threshold: 1024, // Compress responses larger than 1KB
    filter: (req, res) => {
      if (req.headers["x-no-compression"]) {
        return false; // Skip compression if client requests it
      }
      return compression.filter(req, res);
    },
  }),
);

app.get("/api/data", (req, res) => {
  const data = {
    timestamp: new Date().toISOString(),
    items: Array.from({ length: 1000 }, (_, i) => ({
      id: i,
      value: `Item ${i}`,
      metadata: `Additional data for item ${i}`,
    })),
  };

  res.json(data);
});

app.listen(3000, () => {
  console.log("Server running on port 3000");
});

Server Configuration for Compression#

Once your code is ready, configure your server to handle compressed responses.

Nginx#

To enable gzip and Brotli in Nginx, add these directives to your server block:

server {
    listen 80;
    server_name api.example.com;

    # Enable gzip compression
    gzip on;
    gzip_vary on;
    gzip_min_length 1024;
    gzip_comp_level 6;
    gzip_types
        text/plain
        text/css
        text/xml
        text/javascript
        application/json
        application/javascript
        application/xml+rss
        application/atom+xml;

    # Enable Brotli compression
    brotli on;
    brotli_comp_level 6;
    brotli_min_length 1024;
    brotli_types
        text/plain
        text/css
        application/json
        application/javascript
        text/xml
        application/xml
        application/xml+rss;

    location /api/ {
        proxy_pass http://backend;
    }
}

Apache#

For Apache, use mod_deflate for gzip and mod_brotli for Brotli. Add these to your configuration:

# Enable gzip compression
<IfModule mod_deflate.c>
    SetOutputFilter DEFLATE
    SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png|zip|gz|bz2)$ no-gzip dont-vary
    SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary

    AddOutputFilterByType DEFLATE text/plain
    AddOutputFilterByType DEFLATE text/html
    AddOutputFilterByType DEFLATE text/xml
    AddOutputFilterByType DEFLATE text/css
    AddOutputFilterByType DEFLATE application/xml
    AddOutputFilterByType DEFLATE application/xhtml+xml
    AddOutputFilterByType DEFLATE application/rss+xml
    AddOutputFilterByType DEFLATE application/javascript
    AddOutputFilterByType DEFLATE application/json

    DeflateCompressionLevel 6
</IfModule>

# Enable Brotli compression
<IfModule mod_brotli.c>
    BrotliCompressionQuality 6
    BrotliFilterNote Input instream
    BrotliFilterNote Output outstream
    BrotliFilterNote Ratio ratio

    LogFormat '"%r" %{outstream}n/%{instream}n (%{ratio}n%%)' brotli

    AddOutputFilterByType BROTLI_COMPRESS text/plain
    AddOutputFilterByType BROTLI_COMPRESS text/css
    AddOutputFilterByType BROTLI_COMPRESS application/json
    AddOutputFilterByType BROTLI_COMPRESS application/javascript
</IfModule>

Testing Your Implementation#

To confirm that compression is working, use curl to check response headers:

# Test gzip compression
curl -H "Accept-Encoding: gzip" -v https://api.example.com/data

# Test Brotli compression
curl -H "Accept-Encoding: br" -v https://api.example.com/data

# Test both encodings
curl -H "Accept-Encoding: gzip, br" -v https://api.example.com/data

Look for the Content-Encoding header in the response. It should indicate gzip or br, and the Content-Length should reflect the compressed size. This confirms that your API is serving compressed data correctly.

Best Practices for API Compression#

After enabling compression, it's essential to follow some practical guidelines to avoid common issues and ensure your API performs efficiently.

Common Compression Mistakes to Avoid#

Skip compressing non-text content. Compression is most effective for text-based formats like JSON, XML, HTML, CSS, and JavaScript. Media files such as images, videos, PDFs, and ZIP archives are typically already compressed. Trying to compress these again could waste CPU resources or even increase file sizes.

Don't compress very small responses. For tiny payloads, the effort required to compress and decompress data may outweigh the benefits. Set a minimum threshold for response sizes to determine when compression should be applied.

Avoid using maximum compression settings. While high compression levels might slightly reduce file sizes, they often come with a steep cost in CPU usage. Moderate settings usually strike a better balance between performance and efficiency.

Beware of double compression. Ensure data is compressed only once. If your application compresses data before passing it to a web server that also applies compression, it could result in redundant processing or even larger payloads.

Check client 'Accept-Encoding' support. Always verify the "Accept-Encoding" header from clients before applying compression. Some older clients or specialized tools may not support certain compression methods, and sending them compressed data could break functionality.

These precautions can help you avoid unnecessary overhead and ensure smooth operation.

Performance and Resource Monitoring#

To get the most out of compression, keep a close eye on your API's performance metrics.

Track compression effectiveness by endpoint. The impact of compression can vary depending on the type of data an endpoint serves. Text-heavy responses often see significant size reductions, while encrypted or randomized data may not benefit much. Use this information to fine-tune your compression strategy.

Monitor CPU and memory usage. Compression trades CPU and memory for reduced bandwidth. Keep an eye on how different compression settings affect your server's performance, especially during periods of high traffic.

Measure actual performance impact. Compare response times before and after enabling compression. While compressed responses are faster to transmit, the compression and decompression processes can influence overall response times. Use monitoring tools to assess the trade-offs.

Simulate real-world network conditions. Compression is particularly useful on slower connections. Use browser developer tools or network simulation tools to test your API's performance under different connection speeds.

Regular monitoring helps maintain a balance between efficiency and performance.

Security Considerations for Compression#

Mitigate BREACH attack risks. BREACH attacks exploit compression to infer sensitive information from HTTPS responses by analyzing size variations. If your API combines user-controlled input with sensitive data, consider disabling compression for those endpoints.

Separate sensitive data from user input. Design your API to keep sensitive information - like API keys, tokens, or personal data - separate from user-generated content. This reduces the risk of exposing sensitive information.

Add random padding for sensitive responses. Introducing random data to sensitive responses can obscure size patterns, making it harder for attackers to exploit compression vulnerabilities. Use this approach selectively, as it reduces compression efficiency.

Enforce rate limiting. By limiting the number of requests a client can make, you can reduce the likelihood of an attacker exploiting compression vulnerabilities through repeated requests.

Implement strong authentication and CSRF protection. Robust authentication mechanisms and CSRF tokens prevent unauthorized or manipulated requests, which are often used in compression-based attacks.

Monitor for unusual activity. Watch for patterns like repeated requests with minor variations, which could indicate an attempt to exploit compression vulnerabilities.

Conclusion#

Implementing gzip and Brotli compression is a practical way to boost the performance of your REST API. By reducing the size of text payloads, these methods help achieve faster response times, lower bandwidth usage, and a smoother experience for users.

When deciding between the two, gzip is your go-to for broad compatibility, while Brotli offers better compression rates for those seeking maximum efficiency. Both options fit seamlessly into REST API workflows, but the key is knowing which one suits your specific needs and how to apply it effectively.

Focus on compressing text-based data like JSON, while skipping already-compressed files or very small payloads where compression adds little value. Use size thresholds wisely and always respect client capabilities by correctly handling the Accept-Encoding header.

To keep your compression strategy on track, monitor performance regularly and implement robust security practices. Performance tracking allows you to fine-tune settings for the best results, while security measures ensure that vulnerabilities are addressed without compromising the advantages of compression.

For API developers, integrating gzip and Brotli compression can be one of the most impactful ways to optimize performance. The benefits - reduced bandwidth, quicker load times, and happier users - make compression an essential tool in modern API development.

FAQs#

How can I choose between gzip and Brotli for compressing data in my REST API?#

Choosing between gzip and Brotli comes down to your specific needs and priorities:

• Brotli generally offers smaller file sizes thanks to its superior compression ratios, which can lead to faster data transfer. However, achieving higher compression levels might require slightly more processing time.
• Gzip is widely supported by older browsers and servers, making it a safer bet for compatibility. That said, Brotli support has grown significantly in recent years and is now common among modern systems.
• For static assets (like files that are pre-compressed), Brotli tends to be the better option. On the other hand, for dynamic content, gzip often performs better because of its quicker real-time compression.

If your users primarily rely on modern browsers and you’re aiming for top-notch performance, Brotli is a strong choice. But if you need to accommodate older systems or a mix of environments, gzip provides reliable compatibility while still offering solid compression.

What are the performance trade-offs of using data compression in REST APIs?#

Using data compression methods like gzip and Brotli in REST APIs can make data transfer more efficient, but they come with certain trade-offs. For instance, compression increases CPU usage on the server. Brotli, in particular, demands more processing power compared to gzip or serving raw data, which could result in slower response times if server resources are stretched thin.

On the client side, decompressing the data can cause minor delays, especially on devices with limited processing power. Although Brotli delivers better compression ratios than gzip, it’s more resource-heavy, making it essential to weigh the size of the data against the server and client capabilities.

Striking the right balance between performance and efficiency requires careful testing. Experiment with different configurations to identify the best compromise between compression speed and data size reduction for your specific needs.

How can I protect my API's compression setup from vulnerabilities like the BREACH attack?#

To protect your API from vulnerabilities like the BREACH attack, it's crucial to avoid using HTTP compression on endpoints that process sensitive data. Why? Because compression can be exploited to expose confidential information.

If turning off compression entirely isn't an option, here are some practical steps you can take:

• Keep sensitive data separate: Avoid mixing secrets or tokens with user-controlled input.
• Implement CSRF tokens: These can reduce the risk of side-channel attacks.
• Restrict compression to non-sensitive data: Apply compression only where sensitive information isn't involved.

Disabling gzip or Brotli compression for sensitive endpoints remains the most effective way to counter BREACH-related risks. By adopting these measures, you can strike a balance between API performance and security.