Deploying the LiteLLM Gateway: From One pip Command to Virtual Keys, Budgets and Automatic Fallback

First, separate the SDK from the Proxy

Two different things ship under the LiteLLM name, and people mix them up immediately:

• The Python SDK: from litellm import completion. One calling convention for a hundred-plus providers, running inside your process, no extra service. Good for a single script or a single app.
• The Proxy Server: a standalone process listening on port 4000 by default, exposing an OpenAI-compatible API. Every client knows only that one address; key governance, metering, rate limiting, logging and routing all happen there.

The rule of thumb is simple: one caller means the SDK, two or more callers mean the Proxy. This article is about the Proxy.

Figure 1: Where the proxy sits. Downstream only needs one base URL and one gateway-issued key; the upstream providers' original keys exist in exactly one place.

A working gateway in ten minutes

Install

The README now recommends uv, but pip works just as well and lines up more easily with an existing project environment:

python3.12 -m venv .venv
./.venv/bin/pip install 'litellm[proxy]'
./.venv/bin/litellm --version

Check the version afterwards rather than trusting the docs:

LiteLLM: Current Version = 1.99.0

If you just want a quick look, one command starts it: litellm --model gpt-4o. But that gives you no config file and nothing to tune, so move to the config-file route as soon as you have looked.

Write the smallest config that works

Create config.yaml. This version deliberately contains only the essentials:

model_list:
  - model_name: qwen-local
    litellm_params:
      model: ollama_chat/qwen3.5:9b
      api_base: http://127.0.0.1:11434

  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  drop_params: true
  request_timeout: 120

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Three things are worth spelling out:

• `model_name` is an alias; `litellm_params.model` is the real routing target. Downstream only knows the alias, so switching provider or model version is a one-line change with zero client edits.
• `os.environ/XXX` is LiteLLM's own lookup syntax, not shell expansion. Keep secrets out of the YAML — that file is probably going into Git.
• Leave `drop_params: true` on by default. Providers accept different parameters; with this switch on, a field the upstream does not recognise gets dropped instead of failing the whole request.

Start it and verify

LITELLM_MASTER_KEY=sk-demo-1234 \
OPENAI_API_KEY=sk-placeholder \
litellm --config config.yaml --port 4000

The tail of the startup log lists the aliases it loaded — that is your first check:

LiteLLM: Proxy initialized with Config, Set models:
    qwen-local
    gpt-4o

Then send two requests. Health check and model list first:

curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:4000/health/liveliness
curl -s http://127.0.0.1:4000/v1/models -H "Authorization: Bearer sk-demo-1234"

Then a real completion, to confirm it is actually forwarding rather than returning an empty shell:

curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-demo-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-local",
    "messages": [{"role": "user", "content": "Explain what an API gateway does in one sentence"}],
    "max_tokens": 800,
    "reasoning_effort": "none"
  }'

Local qwen3.5:9b answered in one sentence — a gateway is the single entry point that centralises authentication, rate limiting, routing and monitoring across services. Usage came back as 19 input tokens and 42 output.

One small trap when smoke-testing with a reasoning model: without reasoning_effort: "none", the model spends the entire max_tokens budget on its chain of thought, content comes back as an empty string, and everything lands in reasoning_content. The first time you see it, it looks exactly like "the gateway is not connected."

View full-size image

×

The Swagger documentation page LiteLLM serves at its root path, listing OpenAI-compatible endpoints such as chat/completions, embeddings and models

Figure 2: Open http://127.0.0.1:4000/ in a browser and you get auto-generated Swagger docs. Faster than the docs site when you are trying to remember what an endpoint is called.

Making the config production-shaped

The minimal config runs, but it is not finished. Anything you intend to keep running needs at least three more things.

Several deployments behind one alias

The same model_name can appear multiple times. LiteLLM treats those entries as deployments of one model group and load-balances and fails over between them:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 60
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o-eastus
      api_base: os.environ/AZURE_API_BASE
      api_key: os.environ/AZURE_API_KEY
      api_version: "2024-10-21"
      rpm: 120

rpm is not only a limit — it is also what usage-based-routing-v2 uses to split traffic. With 60 on one deployment and 120 on the other, load tilts proportionally toward the one with more headroom.

Routing strategy and cross-model fallback

router_settings:
  routing_strategy: usage-based-routing-v2
  num_retries: 2
  allowed_fails: 3
  cooldown_time: 30
  fallbacks:
    - gpt-4o: ["claude-sonnet", "qwen-local"]

Read that as: retry twice inside the gpt-4o group; take a deployment out of rotation for 30 seconds after three consecutive failures; if the whole group is unusable, drop to claude-sonnet, then to local qwen-local. The value of model routing shows up in that last hop — when every cloud provider is down, you can still return something.

Actually break it once

Writing the config is not the same as it working. Replace both cloud keys with invalid values, request gpt-4o, and read the response headers:

curl -s -D - -o /dev/null http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-demo-1234" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Reply with two words: hello there"}],"max_tokens":200,"reasoning_effort":"none"}'

What came back here:

HTTP/1.1 200 OK
x-litellm-model-name: ollama_chat/qwen3.5:9b
x-litellm-model-group: qwen-local
x-litellm-model-api-base: http://127.0.0.1:11434
x-litellm-version: 1.99.0
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 2
x-litellm-response-duration-ms: 437.109
x-litellm-overhead-duration-ms: 6.084

The caller asked for gpt-4o; local qwen answered. attempted-fallbacks: 2 means the chain walked both hops. *That block of `x-litellm-` headers is the most useful debugging surface here** — faster than reading logs. The gateway's own overhead is six milliseconds; everything else is upstream latency, so it is obvious who is slow.

Turning on the admin UI: virtual keys, budgets and logs

So far the gateway has exactly one master key, shared by everyone, with no way to attribute spend or revoke access individually. Fixing that means attaching PostgreSQL.

The trap you only hit by installing it

Install pip install 'litellm[proxy]', set DATABASE_URL, start it, and 1.99.0 crashes outright:

LiteLLM Proxy:ERROR - Failed to import Prisma client: No module named 'prisma'
Exception: Unable to find Prisma binaries. Please run 'prisma generate' first.
ERROR:    Application startup failed. Exiting.

The [proxy] extra does not include prisma. You have to add it yourself, and prisma generate must run inside the installed litellm package directory, where schema.prisma lives:

pip install prisma
cd .venv/lib/python3.12/site-packages/litellm/proxy
DATABASE_URL="postgresql://user@127.0.0.1:5432/litellm" prisma generate

There is a finer point too: prisma generate shells out to a prisma-client-py executable. If your virtualenv's bin is not on PATH, you get prisma-client-py: command not found rather than a missing-dependency error. Add the venv bin to PATH and rerun. Anyone using the official Docker image meets neither problem — the image is already generated. Which is precisely the argument for using the image if you would rather not wrangle Python environments.

Start again and the migrations run themselves:

All migrations have been successfully applied.
prisma migrate deploy completed

Issue a key with a budget

Everything the admin UI does is also available over the API, which is easier to script:

curl -s -X POST http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer sk-demo-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "key_alias": "frontend-demo",
    "models": ["qwen-local"],
    "max_budget": 5,
    "budget_duration": "30d",
    "rpm_limit": 20,
    "metadata": {"team": "web"}
  }'

The response contains a new sk- key. Calling an allowed model works; calling one it was not granted is refused on the spot:

{"error":{"message":"key not allowed to access model. This key can only access models=['qwen-local']. Tried to access claude-sonnet","type":"key_model_access_denied","param":"model","code":"403"}}

That is the whole point of virtual API keys: one per app or per person, each with its own model allowlist, budget and rate limit, revocable on its own without touching anyone else.

View full-size image

×

The Virtual Keys page of the LiteLLM admin UI showing a key named frontend-demo with spend of 0.00 out of a 5 dollar budget, plus its creation date and budget reset date

Figure 3: The Virtual Keys list. The key created via the API a moment ago appears here with its remaining budget and reset date; you can also create keys straight from the UI.

The admin UI lives at http://127.0.0.1:4000/ui. The default username is admin and the password is the master key; set UI_USERNAME and UI_PASSWORD at startup to change that.

The logs page is where debugging happens

After a dozen or so requests, the Logs page shows duration, TTFT, key alias and status for each one:

View full-size image

×

The Request Logs page of the LiteLLM admin UI listing a dozen requests in reverse chronological order, mostly Success with one Failure, with the Key Alias column showing frontend-demo and litellm_proxy

Figure 4: Request logs. The Failure row in the middle is the unauthorised call from earlier — permission rejections are logged too, which matters a lot when someone reports "it won't connect."

One thing to expect in advance: the Cost column may stay empty. LiteLLM ships a built-in price map; local models and custom deployments are not in it, and startup emits a pile of warnings like this:

register_model: model=azure/gpt-4o-eastus not in built-in cost map and no prefix/region variant matched

Those models are therefore costed at zero. To make cost tracking meaningful, write prices into model_info in your model_list. Do not read a zero in the log as "this was free."

View full-size image

×

The Models and Endpoints page of the LiteLLM admin UI, a table of the aliases loaded from the config file with their upstream deployments and providers

Figure 5: Models + Endpoints. Multiple deployments under one alias are listed on separate rows, which makes "what does this alias actually point at" clearer than reading the YAML.

Six things that catch people out

1. Do not hand the master key to applications. It is an admin credential: it can mint keys and change configuration. Applications get virtual keys.
2. `STORE_MODEL_IN_DB=True` fights with your config file. With it on, models added through the UI live in the database and survive restarts without being overwritten by the YAML. Editing both sides produces the classic "I deleted it from the config but it's still there." Pick one source of truth.
3. Pin secrets like `JWT_SECRET`. Let the process generate one on each boot and a single restart logs everybody out.
4. Set timeouts for your slowest model. The default request_timeout is tight for reasoning models; a long chain of thought gets cut off by your own gateway and looks like an upstream timeout.
5. Probe `/health/liveliness`, not `/`. The root path is the Swagger page and returns 200 no matter what, so using it as a health check is the same as having none.
6. A zero in the Cost column does not mean free — see above.

Who should use it, who can skip it

A good fit: anyone with two or more model providers, anyone who needs to hand access to several applications or colleagues, anyone attributing cost by team, or anyone who wants a local model behind the cloud ones as a last resort. LiteLLM's strength is expressive configuration — routing strategy, retries, fallbacks and budgets all fit in a single YAML file.

Can skip it: a single application calling a single provider is better off with the vendor SDK; a gateway is just one more thing to fail. If all you want is a chat interface, a front end such as Open WebUI (covered in the relay station guide) is more direct.

Worth knowing: LiteLLM moves fast — the version number can jump several patch releases in a day. Pin an exact version in production, never latest, and do not assume a config snippet from six months ago still runs as written. The prisma trap above is exactly that kind of version-dependent breakage.

Alternatives

• The official Docker image, ghcr.io/berriai/litellm, skips the Python environment and the prisma dance entirely. Prefer it for team deployments.
• New API / One API: Chinese-language UI with built-in top-ups and reselling, suited to opening accounts for external users; less expressive configuration than LiteLLM. See the relay station guide.
• [OpenRouter](/tools/openrouter): if you want to maintain nothing at all and accept sending requests through a third party, this is the easiest hosted option.
• AI plugins for Kong or Higress: if you already run one of those API gateways, adding a plugin beats introducing another service.

Sources and how they were checked

• BerriAI/LiteLLM repository — feature descriptions, install methods and configuration fields.
• LiteLLM official documentation — defined behaviour of the Proxy, Router and virtual keys.
• PyPI · litellm — version numbers and release dates; 1.99.0 (2026-09-01) was the latest when this was checked.

The command output, response headers and screenshots in this article all come from a single deployment run on 2026-09-04. Version numbers, the coverage of the built-in price map, and the dependency list of the `[proxy]` extra are all volatile and change with releases; run litellm --version and check which build you actually have before copying anything here.