How to Build Your Own AI API Gateway: Managing AI APIs with New API, One API, and LiteLLM

What Exactly Is a Relay Station?

Many people searching for “how to set up a relay station” are thinking, “I want all my AI tools to use the same interface.” That understanding is correct, but incomplete.

A compliant AI API relay station is more like your own model service front desk: it connects upstream to OpenAI, Anthropic, Google, DeepSeek, Tongyi Qianwen, local models, or other services you legally hold access to; and provides a unified entry point downstream for your applications, scripts, automated workflows, chat frontends, or team members.

It solves not “whether there are models,” but “how to manage models”:

• API Keys for multiple upstream services are no longer scattered across every client.
• Downstream tools only need to remember one Base URL and one downstream token.
• Administrators can see who is calling, which model is being used, and how many tokens are consumed.
• When a channel fails or costs become too high, you can switch, degrade, or restrict it.

If you view each model provider as a water pipe, the relay station is the main utility valve: unified entry, unified metering, unified permissions, and unified logging.

When It Makes Sense to Build Your Own

If you only chat with AI occasionally in your browser, there’s no need to set up a relay server right away. It is truly suited for these scenarios:

1. You use multiple AI tools that all require an OpenAI-compatible API.
2. You work with multiple model providers and want a unified entry point and usage statistics.
3. You don’t want to paste upstream raw keys into every client application.
4. You need to allocate quotas, model whitelists, and expiration times for different tools or team members.
5. You want to connect Open WebUI, coding assistants, automation scripts, or internal systems to the same model backend.
6. You want cheap models to handle simple tasks, powerful models to handle complex tasks, and local models to process private data.

The cases where you should not build one are also clear: if you lack basic operational capabilities, don’t know how to secure your backend, don’t have legitimate upstream sources, plan to publicly sell quotas for unknown models, or cannot bear the responsibilities of data security and compliance.

Core Value: Unification, Cost Reduction, Security

First is unified access. You don’t need to enter different vendors’ API keys in each tool, nor do you need to switch back and forth between OpenAI, Anthropic, Google, and domestic model platforms. Downstream clients only need to remember one entry point; the relay server handles model selection, channel switching, and permission control behind the scenes.

Second is cost efficiency. The relay server can assign simple tasks like summarization, classification, and rewriting to low-cost models, while sending long documents, complex reasoning, and code analysis to more powerful models. When used by a team, you can also see who is spending money and on which models.

Third is security and control. Real upstream keys are no longer scattered across Cursor, Open WebUI, scripts, plugins, and temporary demos; they reside only in the backend. Downstream tools receive tokens you have distributed. If issues arise, you can disable tokens, restrict IPs, view logs, and trace call records.

A relay server isn’t about “hiding away”; it’s about “managing effectively.”

Technical Architecture: Three Layers Are Enough to Understand

A mature AI API relay station can typically be broken down into three layers.

Protocol Conversion Layer. Native APIs from different vendors are not entirely identical; paths, parameters, streaming output, tool-calling formats, and error responses all vary. The relay station attempts to convert them into a unified OpenAI-compatible interface, allowing clients to integrate with just one standard.

Model Routing Layer. It distributes requests to appropriate backend services based on channel priority, weight, model type, group permissions, quotas, health status, and failure retries. More engineering-focused gateways also handle fallbacks, budget limits, caching, and alerts.

Security Management Layer. This layer handles authentication, authorization, rate limiting, auditing, key isolation, access control, and logging. Without this layer, the relay station is merely a forwarder; with it, it becomes a manageable AI gateway.

For personal use, you don’t necessarily need to make it overly complex at first, but you should at least have: downstream tokens, usage statistics, logs, quota limits, HTTPS, backups, and backend access control.

How to Choose an Open-Source Solution

Current common choices can be divided into five categories.

[New API](https://github.com/QuantumNous/new-api): Suitable for long-term personal use or small teams. Evolved from the One API concept, it features a more modern interface with comprehensive management of channels, tokens, model restrictions, usage statistics, groups, and multi-vendor support. It feels more like a ready-made backend system, ideal for those who don’t want to write extensive configurations.

[One API](https://github.com/songquanpeng/one-api): Suitable for quickly understanding the minimal closed loop of a relay station. It is older and lighter, with a direct positioning: accessing multiple models via standard interfaces and managing keys and interface distribution. It’s suitable for getting started first to understand the basic logic of “channels, tokens, logs, and unified Base URLs.”

[LiteLLM Proxy](https://github.com/BerriAI/litellm): Suitable for developers and internal engineering integration. It functions more like a configurable LLM gateway, allowing model mapping, fallbacks, budgets, team management, and unified proxying via config.yaml. While it has a higher learning curve, it fits well into codebases, automated workflows, and internal systems.

[Open WebUI](https://github.com/open-webui/open-webui): Not the relay station itself, but the frontend experience layer. It connects to OpenAI-compatible APIs, enabling regular users to access backend models through a chat interface. You can connect Open WebUI behind New API or LiteLLM to form a personal AI platform.

[sub2api](https://github.com/Wei-Shaw/sub2api): Only recommended for personal research and self-use testing. It leans toward organizing the capabilities of certain subscription-based AI products into an API-style entry point, which can be convenient for specific toolchains but has sensitive boundaries. Do not use it for public operations, shared sales, or bypassing platform rules.

You can judge your choice as follows:

• Want a backend up and running fastest? Prioritize New API.
• Want to understand the basic logic? One API.
• Want engineering-grade routing and fallbacks? LiteLLM Proxy.
• Want a team chat interface? Open WebUI + New API or LiteLLM.
• Want to research subscription entry compatibility? Only look at sub2api in low-risk, self-use environments.

Recommended Path: Start with the New API

If I had to recommend just one more reliable path for individuals or small teams, I’d start with the New API.

The reason is simple: it provides a backend, channels, tokens, model limits, logs, and usage statistics. You don’t need to write configuration files from scratch or build your own user system. Get it running first, then gradually add security and operations capabilities.

1. Prepare the Server and Domain

For personal testing, a cloud server or internal machine with 1 CPU core and 2GB RAM is sufficient to run the basic service. For small teams, I recommend starting with 2 CPU cores and 4GB RAM, along with considering a dedicated database, backups, and a reverse proxy.

Minimum preparation checklist:

• A Linux server or internal host.
• Docker and Docker Compose.
• A domain name or fixed access address.
• At least one valid upstream model API Key.
• A reverse proxy and HTTPS certificate.
• An admin-only backend access entry point.

2. Launch New API with Docker

The New API Docker image can be started directly, using a local data directory to persist configuration:

docker run --name new-api -d --restart always \
  -p 3000:3000 \
  -e TZ=Asia/Shanghai \
  -v ./data:/data \
  calciumion/new-api:latest

After starting, visit:

http://服务器IP:3000

The first thing you should do is not expose it to others, but complete these three actions:

1. Change the default administrator password.
2. Add upstream channel keys that you legally possess.
3. Generate a new key in the "Tokens" section for use by downstream tools.

3. Bind a Domain and HTTPS

In production environments, avoid long-term use of bare IPs and plaintext HTTP. You can use Nginx as a reverse proxy:

server {
  listen 443 ssl;
  server_name api.example.com;

  ssl_certificate /path/to/cert.pem;
  ssl_certificate_key /path/to/key.pem;

  location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

If it is for personal use only, add an extra layer of protection to the admin panel: IP whitelisting, VPN, Cloudflare Access, Basic Auth, or intranet access. The management backend should not be exposed directly to the public internet.

Configure Channels

Channels are responsible for connecting to upstream model services. You need to specify the channel type, Base URL, upstream API Key, model list, priority, weight, and group availability.

Don’t add all channels at once. A more stable approach is:

1. Connect to one upstream service you know is stable.
2. Verify connectivity using the backend test tool.
3. Run a real request.
4. Check if logs and usage are normal.
5. Then connect a second channel and test fallback or weight distribution.

The most important aspect of channel configuration isn’t “having many channels,” but rather “clear sources, stable availability, and transparent billing.”

Create Downstream Tokens

Downstream tools should not use upstream Keys directly; instead, they should use tokens generated by New API. Tokens can be split by purpose:

• chat-webui: For Open WebUI.
• coding-assistant: For coding tools.
• workflow-bot: For automation scripts.
• test-demo: For temporary testing.

For each token, it’s best to set:

• A quota limit.
• An expiration time.
• A model whitelist.
• An IP whitelist.
• A note describing its purpose.

This way, if a tool consumes resources abnormally, you can simply disable the corresponding token without having to chase down every client to change the upstream Key.

How to Configure Downstream Tools

For OpenAI-compatible tools, you generally only need two items:

Base URL: https://api.example.com/v1
API Key: sk-你在中转站生成的下游令牌

A test request can be written like this:

curl https://api.example.com/v1/chat/completions \
  -H "Authorization: Bearer sk-xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      { "role": "user", "content": "用一句话解释 API 中转站" }
    ]
  }'

If your tool does not use the OpenAI-compatible protocol—for example, certain Anthropic-style clients—verify whether your relay has implemented the corresponding protocol adaptation. Many issues where “the key is correct but the request fails” are essentially caused by path or protocol mismatches.

One API: A Lightweight Solution

One API is better suited for quickly understanding the basic closed loop of a relay. A typical Docker startup command looks like this:

docker run --name one-api -d --restart always \
  -p 3000:3000 \
  -e TZ=Asia/Shanghai \
  -v /home/ubuntu/data/one-api:/data \
  justsong/one-api

After logging in, you typically follow three steps:

1. Change the default password.
2. Add upstream model services under “Channels.”
3. Generate downstream access keys under “Tokens.”

One API’s advantages are its lightweight nature, abundant tutorials, and clear concepts. It is suitable for personal learning, small-scale self-use forwarding, and understanding the minimal closed loop of a relay. Its disadvantage is that its modern user experience and fine-grained management capabilities are generally less rich than those of New API.

LiteLLM: An Engineered Gateway

LiteLLM is better suited for developers. Its core idea is to map different models to unified names via a configuration file, then expose an OpenAI-compatible proxy.

A simplified configuration might look like this:

model_list:
  - model_name: cheap-summary
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

  - model_name: strong-reasoning
    litellm_params:
      model: openai/gpt-4.1
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

To start it up:

docker run -d --name litellm \
  -p 4000:4000 \
  -e OPENAI_API_KEY="你的上游Key" \
  -e LITELLM_MASTER_KEY="给下游用的主Key" \
  -v $(pwd)/config.yaml:/app/config.yaml \
  docker.litellm.ai/berriai/litellm:main-latest \
  --config /app/config.yaml

Downstream applications can then point to:

http://你的服务器:4000/v1

LiteLLM is ideal for encoding strategies into your configuration, such as routing summaries to cheaper models, complex reasoning to stronger models, failing over to backup models when necessary, and setting monthly budgets per team. It may not be suitable for those who want to avoid configuration entirely, but it is a great fit for engineering teams.

Open WebUI: Turning the Gateway into a Usable Product

Gateways handle API management, while Open WebUI addresses user experience.

A common setup looks like this:

With this setup, regular users see a chat interface, while administrators manage models, tokens, channels, and budgets. Individuals can also set it up this way for personal use, resulting in a more consistent experience than plugging API URLs directly into various clients.

Security Checklist: Must-Dos Before Launch

The two biggest fears when setting up a relay are backend exposure and key leakage. Before going live, verify at least the following items:

• The administrator default password has been changed.
• The admin panel is not exposed to the public internet, or it is protected by an IP whitelist, VPN, or access control.
• HTTPS is enabled in the production environment.
• Upstream raw API keys are configured only by administrators.
• Downstream tools use only the tokens distributed by the relay.
• Each token has a purpose description, quota, expiration time, and model restrictions.
• Logs capture user, model, token count, response time, and source IP.
• Sensitive requests do not enter untrusted upstreams or third-party relays.
• Data directories are backed up regularly.
• Back up before version upgrades, then validate with a canary release.

If you intend to share this with a team, add the following:

• Group-based permissions.
• Token revocation upon employee offboarding.
• Alerts for abnormal consumption.
• Weekly or monthly cost reviews.
• Audit trails for administrator actions.

API gateway security guidelines repeatedly emphasize authentication, authorization, rate limiting, encryption, logging, and monitoring. The same applies to AI relays; the only difference is that the backend has been swapped for model services.

Compliance Boundaries: Don’t Turn Self-Built Solutions into Gray-Market Operations

Building your own relay does not mean you can operate it publicly. Especially when providing generative AI services to an unspecified public, charging for distributed model capabilities, reselling account quotas, or processing user data, you may encounter issues related to platform terms of service, content safety, data compliance, filing requirements, and payment regulations.

A safer principle is:

• Only connect to upstream model services for which you have legal authorization.
• Serve only yourself, your internal team, or explicitly authorized users.
• Avoid packaging “unlimited quotas,” “low-price sharing,” or “workarounds for restrictions.”
• Do not retain unnecessary sensitive request content.
• Clearly define data usage boundaries for team members.
• When handling client information, code repositories, contracts, financial records, or medical data, prioritize official APIs, enterprise compliance solutions, or local models.

The long-term viability of a relay depends on stability, compliance, and trustworthiness, not short-term cost savings.

Frequently Asked Questions

Is running your own relay always cheaper?

Not necessarily. While the software itself may be open source, the true costs include servers, upstream model API calls, maintenance time, security configuration, and incident handling. Its primary value lies in unified management, with cost optimization being a secondary benefit.

Can I open my relay to others?

Technically, yes—but proceed with extreme caution. Public services introduce risks related to compliance, abuse, payments, content safety, data protection, and upstream terms of service. Individuals and small teams are better off starting with a private or internal tool.

Which should I choose: New API, One API, or LiteLLM?

If you want to avoid writing configuration files and prefer a polished admin interface, go with New API. If you want a lightweight way to learn the fundamentals, check out One API. If you’re looking for engineered routing, budget controls, and fallback logic, look into LiteLLM.

Will the relay station peek at your data?

If you deploy and control the logs and database yourself, the risk depends on your configuration and upstream services. If you use a third-party relay, request content will theoretically pass through their platform; do not send sensitive data to untrusted relays.

Why does the downstream tool still report an error after entering the Key?

Common causes include an incorrect Base URL path, protocol mismatch, model name not on the whitelist, unavailable channel, exhausted token quota, or the upstream Key itself being invalid. Check the relay station logs first, then the client error.

Final Thoughts

“Building your own relay station” may appear to be a technical tutorial, but its true core is management. You aren’t building a secret tunnel; you’re creating a controllable entry point for your AI API calls.

For personal use, the goal is to minimize code changes, reduce key exposure, and maintain visibility into usage. For team use, the goals are clear permissions, cost control, and auditable logs. When dealing with public services, legal compliance, filing requirements, data security, and platform terms must take precedence over technical considerations.

Getting it running is only the first step. The truly valuable relay station is one that operates long-term in a compliant, secure, and stable manner.

Sources

• New API Docker Image
• New API GitHub
• One API GitHub
• LiteLLM GitHub
• LiteLLM Proxy Documentation
• Open WebUI Documentation: Connecting to OpenAI-Compatible APIs
• Open WebUI GitHub
• sub2api GitHub
• MuleSoft: API Gateway Security Best Practices
• AWS: Security Best Practices in Amazon API Gateway

Further Reading

• API Relay
• API Key
• AI Gateway
• Guide to Avoiding Pitfalls in AI Relay Stations