AI Week: Using AI Gateway with the LangChain SDK

Managing API keys for OpenAI and other inference providers creates operational challenges for AI teams. Production keys typically lack visibility into usage and costs, have no built-in rate or spending limits, and can't be easily revoked or scoped to specific applications.

Zuplo's AI Gateway addresses these issues by proxying requests through managed API keys that include usage tracking, cost controls, semantic caching, and security policies.

You can check out John's full application code from this video on GitHub.

The Problem#

A typical LangChain powered application using OpenAI directly will work perfectly fine but does have several limitations:

No usage guardrails on production API keys
Difficult to track costs per application or team
No way to set rate or spending limits
Limited visibility into API usage patterns

How to implement AI Gateway in LangChain#

First, you'll need to set up a Zuplo account if you don't already have one. You can sign up and use the AI Gateway for free.

Then follow the set up guide to create a new team, an OpenAI provider, and an app. You can find all the details on how to do this in the AI Gateway documentation.

Now, with just two small changes, you can route your LangChain application through Zuplo AI Gateway:

Replace your OpenAI API key (API_KEY) with a Zuplo AI Gateway key
Update the BASE_URL to point to your AI Gateway endpoint

def init_chat_model():
    """Initialize the ChatOpenAI model"""
    api_key = os.getenv("API_KEY")
    if not api_key:
        print("❌ Error: Please set your API_KEY in a .env file")
        exit(1)

    # Check for custom BASE_URL - this would be an AI Gateway URL from Zuplo
    base_url = os.getenv("BASE_URL")

    if base_url:
        return ChatOpenAI(api_key=api_key, model="gpt-3.5-turbo", base_url=base_url)
    else:
        return ChatOpenAI(api_key=api_key, model="gpt-3.5-turbo")

In the example code above, when init_chat_model() runs it will check for the presence of a BASE_URL environment variable. If one is found, then LangChain will default to using this instead, routing all requests to the OpenAI gpt-3.5-turbo model through the Zuplo AI Gateway instead of directly to the OpenAI API.

What are the benefits?#

Once configured, you immediately gain:

Usage metrics and dashboards: Track requests, costs, and performance per application
Access control: Revoke keys or set limits per app and team
Semantic caching: Reduce costs and latency for repeated queries
AI firewall: Add security policies without code changes
Cost controls: Set spending limits to prevent budget overruns

Because Zuplo's AI Gateway conforms to OpenAI's API specification, it works seamlessly with LangChain. No need to change providers or modify your application logic.

That's it. Your LangChain application now has production-grade API management with full visibility and control.

For more details on setup and implementation, see the AI Gateway documentation.

AI Week: Using AI Gateway with the LangChain SDK

The Problem#

How to implement AI Gateway in LangChain#

What are the benefits?#

More from AI Week#

Questions? Let's chat

Recent Articles