Designing an AI API Gateway from Scratch: Keys, Rate Limiting, Logging, Billing, and Model Routing

Overall Data Flow

First, visualize the complete journey of a single request through the gateway; each subsequent module will map to this flow:

Application
 | virtual key, OpenAI-compatible format
 v
[1 Auth]     validate the virtual key, load its app and quota
 v
[2 Limit]    check whether the app still has token budget
 v
[3 Route]    pick the upstream vendor and real key by model name / policy
 v
[4 Forward]  translate to the vendor format and send the request (usually SSE streaming)
 v
[5 Meter]    count tokens while forwarding; the stream end gives the total
 v
[6 Bill]     deduct from the app balance at the model's unit price
 v
[7 Log]      write the audit log asynchronously
 v
Application (receives the streamed answer)

A critical design decision: metering and billing occur at the end of the response stream, while rate-limiting pre-checks happen before the request. This time gap between "allow first, settle later" is a core challenge that every AI gateway must address.

Module One: Request Normalization, Speaking a Unified OpenAI Dialect

Standardize all applications to use the OpenAI-compatible format (/v1/chat/completions), with the gateway handling internal translation into each vendor's specific format. This ensures zero code changes for applications when swapping models. The normalization layer must address three key areas: request body field mapping (Anthropic uses a standalone system field, while Gemini employs a different structure), streaming protocol differences, and unified error codes.

Maintain a model registry as the single source of truth for routing and billing:

const MODELS = {
  "gpt-5":          { provider: "openai",    inPrice: 1.25, outPrice: 10, ctx: 400000 },
  "claude-fable-5": { provider: "anthropic", inPrice: 3,    outPrice: 15, ctx: 200000 },
  "deepseek-v3":    { provider: "deepseek",  inPrice: 0.27, outPrice: 1.1, ctx: 128000 },
  // Prices are US dollars per million tokens, illustrative only — use the vendor's live pricing
};

Do not hard-code time-sensitive data like pricing in your code; instead, store it in a configuration center or database to support hot updates—vendor price adjustments are routine, and changing prices should never require a new release.

Module Two: Key Consolidation—Separating Virtual Keys from Real Keys

This is the security core of the gateway. Vendor-specific real API keys reside only within the gateway (encrypted at rest), while applications receive virtual keys issued by the gateway. Each virtual key binds to a set of metadata:

const virtualKey = {
  key: "sk-gw-xxxx",          // issued to the application
  appId: "team-growth",
  allowedModels: ["gpt-5", "deepseek-v3"],
  dailyBudgetUsd: 50,
  rateLimit: { rpm: 60, tpm: 200000 },
  status: "active",           // can be set to revoked at any time
};

The benefits are immediate: if a single key is compromised, it can be revoked with one click without affecting other applications; you can restrict available models and budgets per application; and all calls can be attributed to specific applications. Rotation of real keys occurs internally within the gateway, remaining transparent to the applications.

Module 3: Rate Limiting and Quotas—By Token, Not Request Count

The cost variance of large model requests is enormous; rate limiting by request count is meaningless. You need two layers:

• Rate Limiting: RPM (requests per minute) and TPM (tokens per minute), implemented with sliding windows or token buckets, storing hot data in Redis;
• Budget Quotas: Daily or monthly token budgets per application/user, rejecting requests once exceeded.

The challenge is that you don't know how many tokens a request will consume before it runs (especially for output). The practical approach is "pre-deduction + settlement": perform a pre-check against the estimated upper limit based on max_tokens before the request (reject immediately if the budget is clearly insufficient), then reconcile the difference after receiving the actual usage from the response. This prevents the vulnerability where a small remaining balance allows an exorbitantly expensive request to go through.

View full-size image

×

Real-time traffic curves on the monitoring dashboard

Rate limiting by token: One request can cost thousands of times more than another; counting requests is meaningless.

Module 4: Streaming Metering—Where Things Most Often Go Wrong

The vast majority of requests are streaming (SSE). The gateway must forward and count simultaneously; waiting for the full response to arrive will destroy user experience with high time-to-first-token. Key design points:

• Act as a "man-in-the-middle" for the stream, reading chunks from upstream and forwarding them to the application while accumulating token counts;
• Input tokens can be calculated at request time (or taken from the usage field in the upstream's first packet); output tokens are counted on the fly. Most vendors provide an accurate value in the final chunk's usage field, which should take priority; fall back to a tokenizer estimate only if unavailable;
• Must handle interruptions: If the application disconnects or the upstream times out, you must account for all consumed resources—a partial request still costs money. This is most frequently overlooked, leading to billing discrepancies.

// Pseudocode: metering while forwarding the stream
let outputTokens = 0;
for await (const chunk of upstreamStream) {
  client.write(chunk);              // forward first, to keep latency low
  outputTokens += countChunk(chunk);
  if (chunk.usage) usage = chunk.usage; // prefer the upstream's exact usage
}
// Settle in finally, whether the stream ends normally or breaks
settle(appId, model, inputTokens, usage?.output ?? outputTokens);

Module 5: Billing and Logging

Billing applies the final usage against unit prices from the model registry: Cost = Input tokens × inPrice + Output tokens × outPrice. If Prompt Caching is used, cached hits have different rates and must be billed separately. Ensure atomicity for ledger entries (deducting balances for concurrent requests to the same application) using database transactions or Redis atomic operations.

Logs fall into two categories: Billing Transactions (must be reliable and reconcilable; write synchronously or via a durable queue) and Audit Logs (request/response content used for compliance investigations; high volume, written asynchronously with sampling enabled, ensuring data masking). Do not mix these in the same table.

View full-size image

×

Engineer reviewing system logs and accounts

Separate billing transactions from audit logs: one requires accuracy for reconciliation, while the other demands full traceability; their natures are fundamentally different.

Module 6: Model Routing

With a unified entry point established, Model Routing acts as a policy layer. From simple to complex:

1. Alias Routing: An application requests default-fast, and the gateway maps it to the currently selected specific model; changing models only requires updating the mapping.
2. Failover: If an upstream returns 5xx errors or times out, automatically retry with a backup provider (watch for idempotency and duplicate billing issues).
3. Canary Releases: Route new models by percentage to observe metrics before full rollout.
4. Cost-Based Routing: Direct requests to different model tiers based on characteristics like length or task type—this offers attractive benefits but risks misclassification; start with manual rules, keep feature flags ready, and avoid jumping straight into "intelligent routing."

Routing strategies should be configurable, support canary releases, and allow rollbacks. Do not hard-code them within forwarding logic.

Implementation Timeline and Common Pitfalls

Don't aim for a perfect solution immediately. The first version should implement only the minimal set of Modules 1, 2, and 5: virtual key authentication, usage tracking, and basic logging. First, ensure that "all traffic passes through the gateway and billing is accurate" works end-to-end. Add routing, caching, and auditing incrementally based on actual requirements.

List of high-frequency pitfalls:

Target Audience and Alternatives

This approach suits teams with special requirements for billing, routing, or compliance, or those building the gateway itself as a product. Most teams should not build their own—open-source solutions like LiteLLM and One API have already perfected these modules; using them directly can save weeks of development time. Building your own is justified only in three scenarios: existing solutions' billing models don't match your business needs (e.g., multi-level settlement by customer), you require strong compliance customization, or the gateway itself is what you intend to sell. Even if building from scratch, it is recommended to first study the source code of existing solutions; their pitfalls serve as your design checklist.

Common Questions

Q: Should semantic caching be included in the first version? A: No. While caching can reduce costs, it introduces complex decisions regarding what constitutes an identical request and how long to retain cached data; furthermore, hit rates are typically low for conversational scenarios. Stabilize the primary pipeline first, then add caching as a separate optimization later.

Q: Won't the gateway itself become a latency bottleneck? A: If designed correctly, the gateway adds only millisecond-level overhead, which is negligible compared to second-scale model generation times. The real concern is ensuring the gateway's availability—as a single point of failure, its stability requirements exceed those of any individual application.

Q: How do you isolate multi-tenancy? A: Virtual keys serve as the isolation boundary. Each key binds an independent model whitelist, budget, rate limits, and log storage space; billing is aggregated by key. Tenants do not share real API keys nor can they view each other's usage metrics.

Summary

The core of an AI gateway lies in correctly implementing a streaming pipeline that follows the principle of "allow first, settle later": pre-checking and rate-limiting before requests are forwarded, measuring with low latency during forwarding, accurately billing upon completion, and ensuring no transactions go unrecorded during anomalies. Virtual keys centralize security; model registries unify routing and billing; layered logging provides observability. Once these elements are clear, whether you build from scratch or adopt an existing solution, you will have full control over this layer of infrastructure.