LiteLLM, for Customer Success

For Customer Success Managers

LiteLLM is a self-hosted AI gateway that sits between a customer's applications and many model providers. Apps send one OpenAI-style request; LiteLLM routes it, tracks spend, and controls who can use which models.

Overview
The whole picture
LiteLLM — one gateway for many models presents authorizes sends request selects model forwards records usage sets budget shows checks Customer's AI Apps chatbots, agents, tools Virtual Keys access with limits LiteLLM Gateway one endpoint for many models Route and Fallback pick a working model Spend and Budgets track and cap cost Guardrails policy checks Model Providers OpenAI, Anthropic, Bedrock, and more Admin View who used what
Actors, inputs, and outputs
Concepts
Processes and services
Stores
Infrastructure
100%
Your role

Why this matters for a Customer Success Manager

  • When a customer says they use LiteLLM, they usually mean a shared gateway, not a Python library inside one app.
  • Ask whether apps call vendors directly or go through the gateway. That changes how you diagnose outages, spend spikes, and model-switch requests.
  • Spend questions belong on virtual keys, teams, and the admin view, not on each application's logs.
  • An Anthropic or OpenAI outage may not be a product incident if fallback is already configured in the gateway.
  • SSO, audit logs, and contracted support are LiteLLM Enterprise add-ons. The open-source gateway already covers unified access, virtual keys, budgets, and routing.

How the pieces fit

  • Apps send one OpenAI-style request to LiteLLM.
  • A virtual key authorizes the call and can carry a budget.
  • LiteLLM routes the call to a model provider and records the spend.

Terms customers may use

  • Proxy, AI gateway, and LiteLLM often mean the same shared service.
  • The Python SDK is a library one app can import. The gateway is what the whole organization calls.
  • OpenAI-compatible means apps keep the OpenAI request shape even when the backend is Anthropic, Bedrock, or Azure.

Why customers use it

  • Stop integrating every vendor API in every app.
  • See and cap AI spend before the invoice arrives.
  • Change models without starting a new application project.