Metadata-Version: 2.4 Name: litellm Version: 1.90.0 Summary: Library to easily interface with LLM API providers Author: BerriAI License-Expression: MIT License-File: LICENSE Requires-Dist: fastuuid>=0.14.0,<1.0 Requires-Dist: httpx>=0.28.0,<1.0 Requires-Dist: openai>=2.20.0,<3.0.0 Requires-Dist: python-dotenv>=1.0.0,<2.0 Requires-Dist: tiktoken>=0.8.0,<1.0 Requires-Dist: importlib-metadata>=8.0.0,<9.0 Requires-Dist: tokenizers>=0.21.0,<1.0 Requires-Dist: click>=8.0.0,<9.0 Requires-Dist: jinja2>=3.1.6,<4.0 Requires-Dist: aiohttp>=3.10,<4.0 Requires-Dist: pydantic>=2.10.0,<3.0.0 Requires-Dist: jsonschema>=4.0.0,<5.0 Requires-Dist: diskcache>=5.6.3,<6.0 ; extra == 'caching' Requires-Dist: rich>=13.9.4,<14.0 ; extra == 'cli' Requires-Dist: pyyaml>=6.0.3,<7.0 ; extra == 'cli' Requires-Dist: requests>=2.32.0,<3.0 ; extra == 'cli' Requires-Dist: prisma>=0.11.0,<1.0 ; extra == 'extra-proxy' Requires-Dist: azure-identity>=1.25.2,<2.0 ; extra == 'extra-proxy' Requires-Dist: azure-keyvault-secrets>=4.10.0,<5.0 ; extra == 'extra-proxy' Requires-Dist: google-cloud-kms>=2.24.2,<3.0 ; extra == 'extra-proxy' Requires-Dist: google-cloud-iam>=2.19.1,<3.0 ; extra == 'extra-proxy' Requires-Dist: resend>=2.23.0,<3.0 ; extra == 'extra-proxy' Requires-Dist: redisvl>=0.4.1,<1.0 ; python_full_version < '3.14' and extra == 'extra-proxy' Requires-Dist: a2a-sdk>=0.3.24,<1.0 ; extra == 'extra-proxy' Requires-Dist: google-cloud-aiplatform>=1.133.0,<2.0 ; extra == 'google' Requires-Dist: grpcio==1.78.0 ; extra == 'grpc' Requires-Dist: mlflow>=3.11.1,<4.0 ; extra == 'mlflow' Requires-Dist: gunicorn>=23.0.0,<24.0 ; extra == 'proxy' Requires-Dist: uvicorn>=0.33.0,<1.0 ; extra == 'proxy' Requires-Dist: granian>=2.7.4,<3.0 ; extra == 'proxy' Requires-Dist: uvloop>=0.21.0,<1.0 ; sys_platform != 'win32' and extra == 'proxy' Requires-Dist: fastapi>=0.136.3,<1.0 ; extra == 'proxy' Requires-Dist: starlette>=1.0.1,<2.0 ; extra == 'proxy' Requires-Dist: backoff>=2.2.1,<3.0 ; extra == 'proxy' Requires-Dist: pyyaml>=6.0.3,<7.0 ; extra == 'proxy' Requires-Dist: rq>=2.7.0,<3.0 ; extra == 'proxy' Requires-Dist: orjson>=3.11.6,<4.0 ; extra == 'proxy' Requires-Dist: apscheduler>=3.11.2,<4.0 ; extra == 'proxy' Requires-Dist: fastapi-sso>=0.19.0,<1.0 ; extra == 'proxy' Requires-Dist: pyjwt>=2.13.0,<3.0 ; extra == 'proxy' Requires-Dist: python-multipart>=0.0.27,<1.0 ; extra == 'proxy' Requires-Dist: cryptography>=46.0.7,<47.0 ; extra == 'proxy' Requires-Dist: pynacl>=1.6.2,<2.0 ; extra == 'proxy' Requires-Dist: websockets>=15.0.1,<16.0 ; extra == 'proxy' Requires-Dist: boto3>=1.43.1,<2.0 ; extra == 'proxy' Requires-Dist: azure-identity>=1.25.2,<2.0 ; extra == 'proxy' Requires-Dist: azure-storage-blob>=12.28.0,<13.0 ; extra == 'proxy' Requires-Dist: mcp>=1.26.0,<2.0 ; extra == 'proxy' Requires-Dist: litellm-proxy-extras==0.4.74 ; extra == 'proxy' Requires-Dist: litellm-enterprise==0.1.43 ; extra == 'proxy' Requires-Dist: restrictedpython>=8.1,<9.0 ; extra == 'proxy' Requires-Dist: rich>=13.9.4,<14.0 ; extra == 'proxy' Requires-Dist: polars>=1.38.1,<2.0 ; extra == 'proxy' Requires-Dist: soundfile>=0.12.1,<1.0 ; extra == 'proxy' Requires-Dist: pyroscope-io>=0.8.16,<1.0 ; sys_platform != 'win32' and extra == 'proxy' Requires-Dist: pydantic-settings>=2.14.1,<3.0 ; extra == 'proxy' Requires-Dist: google-cloud-aiplatform>=1.133.0,<2.0 ; extra == 'proxy-runtime' Requires-Dist: google-genai>=1.37.0,<2.0 ; extra == 'proxy-runtime' Requires-Dist: anthropic[vertex]>=0.84.0,<1.0 ; extra == 'proxy-runtime' Requires-Dist: grpcio==1.78.0 ; extra == 'proxy-runtime' Requires-Dist: prometheus-client>=0.20.0,<1.0 ; extra == 'proxy-runtime' Requires-Dist: langfuse>=2.59.7,<3.0 ; extra == 'proxy-runtime' Requires-Dist: opentelemetry-api==1.28.0 ; extra == 'proxy-runtime' Requires-Dist: opentelemetry-sdk==1.28.0 ; extra == 'proxy-runtime' Requires-Dist: opentelemetry-exporter-otlp==1.28.0 ; extra == 'proxy-runtime' Requires-Dist: opentelemetry-instrumentation-fastapi==0.49b0 ; extra == 'proxy-runtime' Requires-Dist: ddtrace>=2.19.0,<3.0 ; extra == 'proxy-runtime' Requires-Dist: sentry-sdk>=2.21.0,<3.0 ; extra == 'proxy-runtime' Requires-Dist: mangum>=0.17.0,<1.0 ; extra == 'proxy-runtime' Requires-Dist: azure-ai-contentsafety>=1.0.0,<2.0 ; extra == 'proxy-runtime' Requires-Dist: azure-storage-file-datalake>=12.20.0,<13.0 ; extra == 'proxy-runtime' Requires-Dist: pypdf>=6.12.0,<7.0 ; python_full_version < '3.14' and extra == 'proxy-runtime' Requires-Dist: llm-sandbox>=0.3.39,<1.0 ; extra == 'proxy-runtime' Requires-Dist: detect-secrets>=1.5.0,<2.0 ; extra == 'proxy-runtime' Requires-Dist: semantic-router>=0.1.15,<1.0 ; python_full_version < '3.14' and extra == 'semantic-router' Requires-Dist: aurelio-sdk>=0.0.19,<1.0 ; python_full_version < '3.14' and extra == 'semantic-router' Requires-Dist: nvidia-riva-client>=2.15.0 ; extra == 'stt-nvidia-riva' Requires-Dist: soundfile>=0.12.1 ; extra == 'stt-nvidia-riva' Requires-Dist: audioread>=3.0.1 ; extra == 'stt-nvidia-riva' Requires-Dist: numpy>=1.26.0 ; extra == 'stt-nvidia-riva' Requires-Dist: numpydoc>=1.8.0,<2.0 ; extra == 'utils' Requires-Python: >=3.10, <3.14 Project-URL: Homepage, https://litellm.ai Project-URL: Repository, https://github.com/BerriAI/litellm Project-URL: Documentation, https://docs.litellm.ai Provides-Extra: caching Provides-Extra: cli Provides-Extra: extra-proxy Provides-Extra: google Provides-Extra: grpc Provides-Extra: mlflow Provides-Extra: proxy Provides-Extra: proxy-runtime Provides-Extra: semantic-router Provides-Extra: stt-nvidia-riva Provides-Extra: utils Description-Content-Type: text/markdown

🚅 LiteLLM

LiteLLM AI Gateway

Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.

Deploy to Render Deploy on Railway Deploy on AWS Deploy on GCP

LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website

PyPI Version GitHub Stars Y Combinator W23 Whatsapp Discord Slack CodSpeed

LiteLLM AI Gateway --- ## What is LiteLLM LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format. Use it as a **Python SDK** for direct library integration, or deploy the **AI Gateway (Proxy Server)** as a centralized service for your team or organization. [**Jump to LiteLLM Proxy (LLM Gateway) Docs**](https://docs.litellm.ai/docs/simple_proxy)
[**Jump to Supported LLM Providers**](https://docs.litellm.ai/docs/providers) --- ## Why LiteLLM Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction: - **Unified API** — one interface for 100+ LLMs, no provider-specific SDK juggling - **Drop-in OpenAI compatibility** — swap providers without rewriting your code - **Production-ready gateway** — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box - **8ms P95 latency** at 1k RPS ([benchmarks](https://docs.litellm.ai/docs/benchmarks)) ### OSS Adopters
Stripe image Google ADK Greptile OpenHands

Netflix

 OpenAI Agents SDK
--- ## Features
LLMs - Call 100+ LLMs (Python SDK + AI Gateway) [**All Supported Endpoints**](https://docs.litellm.ai/docs/supported_endpoints) - `/chat/completions`, `/responses`, `/embeddings`, `/images`, `/audio`, `/batches`, `/rerank`, `/a2a`, `/messages` and more. ### Python SDK ```shell uv add litellm ``` ```python from litellm import completion import os os.environ["OPENAI_API_KEY"] = "your-openai-key" os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key" # OpenAI response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}]) # Anthropic response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}]) ``` ### AI Gateway (Proxy Server) [**Getting Started - E2E Tutorial**](https://docs.litellm.ai/docs/proxy/docker_quick_start) - Setup virtual keys, make your first request ```shell uv tool install 'litellm[proxy]' litellm --model gpt-4o ``` ```python import openai client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000") response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}] ) ``` [**Docs: LLM Providers**](https://docs.litellm.ai/docs/providers)
Agents - Invoke A2A Agents (Python SDK + AI Gateway) [**Supported Providers**](https://docs.litellm.ai/docs/a2a#add-a2a-agents) - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI ### Python SDK - A2A Protocol ```python from litellm.a2a_protocol import A2AClient from a2a.types import SendMessageRequest, MessageSendParams from uuid import uuid4 client = A2AClient(base_url="http://localhost:10001") request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams( message={ "role": "user", "parts": [{"kind": "text", "text": "Hello!"}], "messageId": uuid4().hex, } ) ) response = await client.send_message(request) ``` ### AI Gateway (Proxy Server) **Step 1.** [Add your Agent to the AI Gateway](https://docs.litellm.ai/docs/a2a#adding-your-agent) **Step 2.** Call Agent via A2A SDK ```python from a2a.client import A2ACardResolver, A2AClient from a2a.types import MessageSendParams, SendMessageRequest from uuid import uuid4 import httpx base_url = "http://localhost:4000/a2a/my-agent" # LiteLLM proxy + agent name headers = {"Authorization": "Bearer sk-1234"} # LiteLLM Virtual Key async with httpx.AsyncClient(headers=headers) as httpx_client: resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url) agent_card = await resolver.get_agent_card() client = A2AClient(httpx_client=httpx_client, agent_card=agent_card) request = SendMessageRequest( id=str(uuid4()), params=MessageSendParams( message={ "role": "user", "parts": [{"kind": "text", "text": "Hello!"}], "messageId": uuid4().hex, } ) ) response = await client.send_message(request) ``` [**Docs: A2A Agent Gateway**](https://docs.litellm.ai/docs/a2a)
MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway) ### Python SDK - MCP Bridge ```python from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio_client from litellm import experimental_mcp_client import litellm server_params = StdioServerParameters(command="python", args=["mcp_server.py"]) async with stdio_client(server_params) as (read, write): async with ClientSession(read, write) as session: await session.initialize() # Load MCP tools in OpenAI format tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai") # Use with any LiteLLM model response = await litellm.acompletion( model="gpt-4o", messages=[{"role": "user", "content": "What's 3 + 5?"}], tools=tools ) ``` ### AI Gateway - MCP Gateway **Step 1.** [Add your MCP Server to the AI Gateway](https://docs.litellm.ai/docs/mcp#adding-your-mcp) **Step 2.** Call MCP tools via `/chat/completions` ```bash curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \ -H 'Authorization: Bearer sk-1234' \ -H 'Content-Type: application/json' \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Summarize the latest open PR"}], "tools": [{ "type": "mcp", "server_url": "litellm_proxy/mcp/github", "server_label": "github_mcp", "require_approval": "never" }] }' ``` ### Use with Cursor IDE ```json { "mcpServers": { "LiteLLM": { "url": "http://localhost:4000/mcp/", "headers": { "x-litellm-api-key": "Bearer sk-1234" } } } } ``` [**Docs: MCP Gateway**](https://docs.litellm.ai/docs/mcp)
### Supported Providers ([Website Supported Models](https://models.litellm.ai/) | [Docs](https://docs.litellm.ai/docs/providers)) | Provider | `/chat/completions` | `/messages` | `/responses` | `/embeddings` | `/image/generations` | `/audio/transcriptions` | `/audio/speech` | `/moderations` | `/batches` | `/rerank` | |-------------------------------------------------------------------------------------|---------------------|-------------|--------------|---------------|----------------------|-------------------------|-----------------|----------------|-----------|-----------| | [Abliteration (`abliteration`)](https://docs.litellm.ai/docs/providers/abliteration) | ✅ | | | | | | | | | | | [AI/ML API (`aiml`)](https://docs.litellm.ai/docs/providers/aiml) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | [AI21 (`ai21`)](https://docs.litellm.ai/docs/providers/ai21) | ✅ | ✅ | ✅ | | | | | | | | | [AI21 Chat (`ai21_chat`)](https://docs.litellm.ai/docs/providers/ai21) | ✅ | ✅ | ✅ | | | | | | | | | [Aleph Alpha](https://docs.litellm.ai/docs/providers/aleph_alpha) | ✅ | ✅ | ✅ | | | | | | | | | [Amazon Nova](https://docs.litellm.ai/docs/providers/amazon_nova) | ✅ | ✅ | ✅ | | | | | | | | | [Anthropic (`anthropic`)](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | ✅ | ✅ | | | | | | ✅ | | | [Anthropic Text (`anthropic_text`)](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | ✅ | ✅ | | | | | | ✅ | | | [Anyscale](https://docs.litellm.ai/docs/providers/anyscale) | ✅ | ✅ | ✅ | | | | | | | | | [AssemblyAI (`assemblyai`)](https://docs.litellm.ai/docs/pass_through/assembly_ai) | ✅ | ✅ | ✅ | | | ✅ | | | | | | [Auto Router (`auto_router`)](https://docs.litellm.ai/docs/proxy/auto_routing) | ✅ | ✅ | ✅ | | | | | | | | | [AWS - Bedrock (`bedrock`)](https://docs.litellm.ai/docs/providers/bedrock) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ | | [AWS - Sagemaker (`sagemaker`)](https://docs.litellm.ai/docs/providers/aws_sagemaker) | ✅ | ✅ | ✅ | ✅ | | | | | | | | [Azure (`azure`)](https://docs.litellm.ai/docs/providers/azure) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | [Azure AI (`azure_ai`)](https://docs.litellm.ai/docs/providers/azure_ai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | [Azure Text (`azure_text`)](https://docs.litellm.ai/docs/providers/azure) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | | | [Baseten (`baseten`)](https://docs.litellm.ai/docs/providers/baseten) | ✅ | ✅ | ✅ | | | | | | | | | [Bytez (`bytez`)](https://docs.litellm.ai/docs/providers/bytez) | ✅ | ✅ | ✅ | | | | | | | | | [Cerebras (`cerebras`)](https://docs.litellm.ai/docs/providers/cerebras) | ✅ | ✅ | ✅ | | | | | | | | | [Clarifai (`clarifai`)](https://docs.litellm.ai/docs/providers/clarifai) | ✅ | ✅ | ✅ | | | | | | | | | [Cloudflare AI Workers (`cloudflare`)](https://docs.litellm.ai/docs/providers/cloudflare_workers) | ✅ | ✅ | ✅ | | | | | | | | | [Codestral (`codestral`)](https://docs.litellm.ai/docs/providers/codestral) | ✅ | ✅ | ✅ | | | | | | | | | [Cohere (`cohere`)](https://docs.litellm.ai/docs/providers/cohere) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ | | [Cohere Chat (`cohere_chat`)](https://docs.litellm.ai/docs/providers/cohere) | ✅ | ✅ | ✅ | | | | | | | | | [CometAPI (`cometapi`)](https://docs.litellm.ai/docs/providers/cometapi) | ✅ | ✅ | ✅ | ✅ | | | | | | | | [CompactifAI (`compactifai`)](https://docs.litellm.ai/docs/providers/compactifai) | ✅ | ✅ | ✅ | | | | | | | | | [Custom (`custom`)](https://docs.litellm.ai/docs/providers/custom_llm_server) | ✅ | ✅ | ✅ | | | | | | | | | [Custom OpenAI (`custom_openai`)](https://docs.litellm.ai/docs/providers/openai_compatible) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | | | [Dashscope (`dashscope`)](https://docs.litellm.ai/docs/providers/dashscope) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ | | [Databricks (`databricks`)](https://docs.litellm.ai/docs/providers/databricks) | ✅ | ✅ | ✅ | | | | | | | | | [DataRobot (`datarobot`)](https://docs.litellm.ai/docs/providers/datarobot) | ✅ | ✅ | ✅ | | | | | | | | | [Deepgram (`deepgram`)](https://docs.litellm.ai/docs/providers/deepgram) | ✅ | ✅ | ✅ | | | ✅ | | | | | | [DeepInfra (`deepinfra`)](https://docs.litellm.ai/docs/providers/deepinfra) | ✅ | ✅ | ✅ | | | | | | | | | [Deepseek (`deepseek`)](https://docs.litellm.ai/docs/providers/deepseek) | ✅ | ✅ | ✅ | | | | | | | | | [ElevenLabs (`elevenlabs`)](https://docs.litellm.ai/docs/providers/elevenlabs) | ✅ | ✅ | ✅ | | | ✅ | ✅ | | | | | [Empower (`empower`)](https://docs.litellm.ai/docs/providers/empower) | ✅ | ✅ | ✅ | | | | | | | | | [Fal AI (`fal_ai`)](https://docs.litellm.ai/docs/providers/fal_ai) | ✅ | ✅ | ✅ | | ✅ | | | | | | | [Featherless AI (`featherless_ai`)](https://docs.litellm.ai/docs/providers/featherless_ai) | ✅ | ✅ | ✅ | | | | | | | | | [Fireworks AI (`fireworks_ai`)](https://docs.litellm.ai/docs/providers/fireworks_ai) | ✅ | ✅ | ✅ | | | | | | | | | [FriendliAI (`friendliai`)](https://docs.litellm.ai/docs/providers/friendliai) | ✅ | ✅ | ✅ | | | | | | | | | [Galadriel (`galadriel`)](https://docs.litellm.ai/docs/providers/galadriel) | ✅ | ✅ | ✅ | | | | | | | | | [GitHub Copilot (`github_copilot`)](https://docs.litellm.ai/docs/providers/github_copilot) | ✅ | ✅ | ✅ | ✅ | | | | | | | | [GitHub Models (`github`)](https://docs.litellm.ai/docs/providers/github) | ✅ | ✅ | ✅ | | | | | | | | | [Google - PaLM](https://docs.litellm.ai/docs/providers/palm) | ✅ | ✅ | ✅ | | | | | | | | | [Google - Vertex AI (`vertex_ai`)](https://docs.litellm.ai/docs/providers/vertex) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | [Google AI Studio - Gemini (`gemini`)](https://docs.litellm.ai/docs/providers/gemini) | ✅ | ✅ | ✅ | | | | | | | | | [GradientAI (`gradient_ai`)](https://docs.litellm.ai/docs/providers/gradient_ai) | ✅ | ✅ | ✅ | | | | | | | | | [Groq AI (`groq`)](https://docs.litellm.ai/docs/providers/groq) | ✅ | ✅ | ✅ | | | | | | | | | [Heroku (`heroku`)](https://docs.litellm.ai/docs/providers/heroku) | ✅ | ✅ | ✅ | | | | | | | | | [Hosted VLLM (`hosted_vllm`)](https://docs.litellm.ai/docs/providers/vllm) | ✅ | ✅ | ✅ | | | | | | | | | [Huggingface (`huggingface`)](https://docs.litellm.ai/docs/providers/huggingface) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ | | [Hyperbolic (`hyperbolic`)](https://docs.litellm.ai/docs/providers/hyperbolic) | ✅ | ✅ | ✅ | | | | | | | | | [IBM - Watsonx.ai (`watsonx`)](https://docs.litellm.ai/docs/providers/watsonx) | ✅ | ✅ | ✅ | ✅ | | | | | | | | [Infinity (`infinity`)](https://docs.litellm.ai/docs/providers/infinity) | | | | ✅ | | | | | | | | [Jina AI (`jina_ai`)](https://docs.litellm.ai/docs/providers/jina_ai) | | | | ✅ | | | | | | | | [Lambda AI (`lambda_ai`)](https://docs.litellm.ai/docs/providers/lambda_ai) | ✅ | ✅ | ✅ | | | | | | | | | [Lemonade (`lemonade`)](https://docs.litellm.ai/docs/providers/lemonade) | ✅ | ✅ | ✅ | | | | | | | | | [LiteLLM Proxy (`litellm_proxy`)](https://docs.litellm.ai/docs/providers/litellm_proxy) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | | [Llamafile (`llamafile`)](https://docs.litellm.ai/docs/providers/llamafile) | ✅ | ✅ | ✅ | | | | | | | | | [LM Studio (`lm_studio`)](https://docs.litellm.ai/docs/providers/lm_studio) | ✅ | ✅ | ✅ | | | | | | | | | [Maritalk (`maritalk`)](https://docs.litellm.ai/docs/providers/maritalk) | ✅ | ✅ | ✅ | | | | | | | | | [Meta - Llama API (`meta_llama`)](https://docs.litellm.ai/docs/providers/meta_llama) | ✅ | ✅ | ✅ | | | | | | | | | [Mistral AI API (`mistral`)](https://docs.litellm.ai/docs/providers/mistral) | ✅ | ✅ | ✅ | ✅ | | | | | | | | [ModelScope (`modelscope`)](https://docs.litellm.ai/docs/providers/modelscope) | ✅ | ✅ | ✅ | | ✅ | | | | | | | [Moonshot (`moonshot`)](https://docs.litellm.ai/docs/providers/moonshot) | ✅ | ✅ | ✅ | | | | | | | | | [Morph (`morph`)](https://docs.litellm.ai/docs/providers/morph) | ✅ | ✅ | ✅ | | | | | | | | | [Nebius AI Studio (`nebius`)](https://docs.litellm.ai/docs/providers/nebius) | ✅ | ✅ | ✅ | ✅ | | | | | | | | [NLP Cloud (`nlp_cloud`)](https://docs.litellm.ai/docs/providers/nlp_cloud) | ✅ | ✅ | ✅ | | | | | | | | | [Novita AI (`novita`)](https://novita.ai/models/llm?utm_source=github_litellm&utm_medium=github_readme&utm_campaign=github_link) | ✅ | ✅ | ✅ | | | | | | | | | [Nscale (`nscale`)](https://docs.litellm.ai/docs/providers/nscale) | ✅ | ✅ | ✅ | | | | | | | | | [Nvidia NIM (`nvidia_nim`)](https://docs.litellm.ai/docs/providers/nvidia_nim) | ✅ | ✅ | ✅ | | | | | | | | | [OCI (`oci`)](https://docs.litellm.ai/docs/providers/oci) | ✅ | ✅ | ✅ | | | | | | | | | [Ollama (`ollama`)](https://docs.litellm.ai/docs/providers/ollama) | ✅ | ✅ | ✅ | ✅ | | | | | | | | [Ollama Chat (`ollama_chat`)](https://docs.litellm.ai/docs/providers/ollama) | ✅ | ✅ | ✅ | | | | | | | | | [Oobabooga (`oobabooga`)](https://docs.litellm.ai/docs/providers/openai_compatible) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | | | [OpenAI (`openai`)](https://docs.litellm.ai/docs/providers/openai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | [OpenAI-like (`openai_like`)](https://docs.litellm.ai/docs/providers/openai_compatible) | | | | ✅ | | | | | | | | [OpenRouter (`openrouter`)](https://docs.litellm.ai/docs/providers/openrouter) | ✅ | ✅ | ✅ | | | | | | | | | [OVHCloud AI Endpoints (`ovhcloud`)](https://docs.litellm.ai/docs/providers/ovhcloud) | ✅ | ✅ | ✅ | | | | | | | | | [Perplexity AI (`perplexity`)](https://docs.litellm.ai/docs/providers/perplexity) | ✅ | ✅ | ✅ | | | | | | | | | [Petals (`petals`)](https://docs.litellm.ai/docs/providers/petals) | ✅ | ✅ | ✅ | | | | | | | | | [Pinstripes (`pinstripes`)](https://docs.litellm.ai/docs/providers/pinstripes) | ✅ | ✅ | ✅ | | | | | | | | | [Predibase (`predibase`)](https://docs.litellm.ai/docs/providers/predibase) | ✅ | ✅ | ✅ | | | | | | | | | [Recraft (`recraft`)](https://docs.litellm.ai/docs/providers/recraft) | | | | | ✅ | | | | | | | [Replicate (`replicate`)](https://docs.litellm.ai/docs/providers/replicate) | ✅ | ✅ | ✅ | | | | | | | | | [Sagemaker Chat (`sagemaker_chat`)](https://docs.litellm.ai/docs/providers/aws_sagemaker) | ✅ | ✅ | ✅ | | | | | | | | | [Sambanova (`sambanova`)](https://docs.litellm.ai/docs/providers/sambanova) | ✅ | ✅ | ✅ | | | | | | | | | [Snowflake (`snowflake`)](https://docs.litellm.ai/docs/providers/snowflake) | ✅ | ✅ | ✅ | | | | | | | | | [Text Completion Codestral (`text-completion-codestral`)](https://docs.litellm.ai/docs/providers/codestral) | ✅ | ✅ | ✅ | | | | | | | | | [Text Completion OpenAI (`text-completion-openai`)](https://docs.litellm.ai/docs/providers/text_completion_openai) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | | | [Together AI (`together_ai`)](https://docs.litellm.ai/docs/providers/togetherai) | ✅ | ✅ | ✅ | | | | | | | | | [Topaz (`topaz`)](https://docs.litellm.ai/docs/providers/topaz) | ✅ | ✅ | ✅ | | | | | | | | | [Triton (`triton`)](https://docs.litellm.ai/docs/providers/triton-inference-server) | ✅ | ✅ | ✅ | | | | | | | | | [V0 (`v0`)](https://docs.litellm.ai/docs/providers/v0) | ✅ | ✅ | ✅ | | | | | | | | | [Vercel AI Gateway (`vercel_ai_gateway`)](https://docs.litellm.ai/docs/providers/vercel_ai_gateway) | ✅ | ✅ | ✅ | | | | | | | | | [VLLM (`vllm`)](https://docs.litellm.ai/docs/providers/vllm) | ✅ | ✅ | ✅ | | | | | | | | | [Volcengine (`volcengine`)](https://docs.litellm.ai/docs/providers/volcano) | ✅ | ✅ | ✅ | | | | | | | | | [Voyage AI (`voyage`)](https://docs.litellm.ai/docs/providers/voyage) | | | | ✅ | | | | | | | | [WandB Inference (`wandb`)](https://docs.litellm.ai/docs/providers/wandb_inference) | ✅ | ✅ | ✅ | | | | | | | | | [Watsonx Text (`watsonx_text`)](https://docs.litellm.ai/docs/providers/watsonx) | ✅ | ✅ | ✅ | | | | | | | | | [xAI (`xai`)](https://docs.litellm.ai/docs/providers/xai) | ✅ | ✅ | ✅ | | | | | | | | | [Xinference (`xinference`)](https://docs.litellm.ai/docs/providers/xinference) | | | | ✅ | | | | | | | [**Read the Docs**](https://docs.litellm.ai/docs/) --- ## Get Started You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:
LiteLLM AI Gateway LiteLLM Python SDK
Use Case Central service (LLM Gateway) to access multiple LLMs Use LiteLLM directly in your Python code
Who Uses It? Gen AI Enablement / ML Platform Teams Developers building LLM projects
Key Features Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.)
**Stable Release:** Use docker images with the `-stable` tag. These have undergone 12 hour load tests, before being published. [More information about the release cycle here](https://docs.litellm.ai/docs/proxy/release_cycle) Support for more providers. Missing a provider or LLM Platform, raise a [feature request](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.yml&title=%5BFeature%5D%3A+). ### Deploy on AWS or GCP with Terraform Run the LiteLLM proxy as a production-ready componentized stack (gateway, backend, UI on separate services; managed Postgres + Redis + object store) using the published Terraform modules. Both modules are on the [public Terraform Registry](https://registry.terraform.io/namespaces/BerriAI) — no auth needed. #### AWS — ECS Fargate + Aurora + ElastiCache + ALB [![Launch in AWS CloudShell](https://img.shields.io/badge/Launch-AWS_CloudShell-FF9900?logo=amazon-aws&logoColor=white)](https://console.aws.amazon.com/cloudshell/home) — opens an in-browser shell, already authenticated to your AWS account. Once inside, run: ```bash git clone https://github.com/BerriAI/litellm.git cd litellm/terraform/litellm/aws/examples/default cp terraform.tfvars.example terraform.tfvars # edit region/tenant/env terraform init && terraform apply ``` [Module page →](https://registry.terraform.io/modules/BerriAI/litellm/aws/latest) Or call the module from your own root config: ```hcl # main.tf terraform { required_version = ">= 1.6.0" required_providers { aws = { source = "hashicorp/aws", version = "~> 5.60" } } } provider "aws" { region = "us-west-2" } module "litellm" { source = "BerriAI/litellm/aws" version = "~> 1.89" region = "us-west-2" azs = ["us-west-2a", "us-west-2b"] tenant = "acme" env = "prod" # Production: provide an ACM cert. Without one, set allow_plaintext_alb = true # (dev/trial only). # acm_certificate_arn = "arn:aws:acm:us-west-2:111122223333:certificate/..." allow_plaintext_alb = true } output "litellm_url" { value = module.litellm.alb_dns_name } ``` ```bash terraform init terraform apply ``` Provider API keys live in AWS Secrets Manager; reference ARNs via `gateway_extra_secrets`. Full input list and architecture diagram on the [registry page](https://registry.terraform.io/modules/BerriAI/litellm/aws/latest?tab=inputs). #### GCP — Cloud Run + Cloud SQL + Memorystore + HTTPS LB [![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.png)](https://ssh.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https%3A%2F%2Fgithub.com%2FBerriAI%2Flitellm&cloudshell_workspace=terraform%2Flitellm%2Fgcp%2Fexamples%2Fdefault&cloudshell_tutorial=TUTORIAL.md&cloudshell_image=gcr.io/ds-artifacts-cloudshell/deploystack_custom_image&shellonly=true) Real 1-click. Opens Cloud Shell, clones this repo, and walks you through `terraform apply` via a built-in [DeployStack tutorial](./terraform/litellm/gcp/examples/default/TUTORIAL.md) — pick the project, the tutorial sets up the Artifact Registry remote repo, writes `terraform.tfvars` from your answers, and runs apply. [Module page →](https://registry.terraform.io/modules/BerriAI/litellm/google/latest) To call the module from your own config instead, Cloud Run can't pull from `ghcr.io` directly, so first set up a one-time Artifact Registry remote repo backed by GHCR: ```bash gcloud artifacts repositories create litellm \ --location=us-central1 \ --repository-format=docker \ --mode=remote-repository \ --remote-docker-repo=https://ghcr.io \ --project=my-gcp-project ``` Then: ```hcl # main.tf terraform { required_version = ">= 1.6.0" required_providers { google = { source = "hashicorp/google", version = "~> 6.10" } google-beta = { source = "hashicorp/google-beta", version = "~> 6.10" } } } provider "google" { project = "my-gcp-project"; region = "us-central1" } provider "google-beta" { project = "my-gcp-project"; region = "us-central1" } module "litellm" { source = "BerriAI/litellm/google" version = "~> 1.89" project_id = "my-gcp-project" region = "us-central1" tenant = "acme" env = "prod" # Replace my-gcp-project with your GCP project ID (same value as project_id above). image_registry = "us-central1-docker.pkg.dev/my-gcp-project/litellm/berriai" # Production: provide DNS already pointing at the LB IP for Google-managed certs. # Without one, set allow_plaintext_lb = true (dev/trial only). # lb_domains = ["proxy.example.com"] allow_plaintext_lb = true } output "litellm_url" { value = module.litellm.load_balancer_url } ``` ```bash terraform init terraform apply ``` Provider API keys live in Secret Manager; reference resource IDs (e.g. `projects/my-gcp-project/secrets/openai-api-key`) via `gateway_extra_secrets`. Full input list and architecture diagram on the [registry page](https://registry.terraform.io/modules/BerriAI/litellm/google/latest?tab=inputs). #### Both stacks include - The full componentized split (gateway / backend / UI as independent services) - Managed Postgres (writer + reader) and Redis - Versioned object store for proxy state + file uploads - An auto-generated `LITELLM_MASTER_KEY` in your cloud's secret manager - A one-off migration job that runs `prisma migrate deploy` before the proxy starts - The same `proxy_config` surface as the [Helm chart](./helm/litellm/) — pass YAML as a typed map The Terraform modules live at [`terraform/litellm/aws/`](./terraform/litellm/aws/) and [`terraform/litellm/gcp/`](./terraform/litellm/gcp/) in this repo; the registry entries are read-only mirrors updated on each release. ### Run in Developer Mode #### Services 1. Setup .env file in root 2. Run dependent services `docker-compose up db prometheus` #### Backend 1. (In root) create virtual environment `python -m venv .venv` 2. Activate virtual environment `source .venv/bin/activate` 3. Install dependencies `uv sync --all-extras --group proxy-dev` 4. `uv run prisma generate` 5. `prisma generate` 6. Start proxy backend `python litellm/proxy/proxy_cli.py` #### Frontend 1. Navigate to `ui/litellm-dashboard` 2. Install dependencies `npm install` 3. Run `npm run dev` to start the dashboard ### Verify Docker Image Signatures All LiteLLM Docker images published to GHCR are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm: ``` **Verify using a release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm//cosign.pub \ ghcr.io/berriai/litellm: ``` Replace `` with the version you are deploying (e.g. `v1.83.0-stable`). --- # Enterprise For companies that need better security, user management and professional support [Get an Enterprise License](https://litellm.ai/enterprise) [Talk to founders](https://enterprise.litellm.ai/demo) This covers: - ✅ **Features under the [LiteLLM Commercial License](https://docs.litellm.ai/docs/proxy/enterprise):** - ✅ **Feature Prioritization** - ✅ **Custom Integrations** - ✅ **Professional Support - Dedicated discord + slack** - ✅ **Custom SLAs** - ✅ **Secure access with Single Sign-On** # Contributing We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help. ## Quick Start for Contributors This requires uv to be installed. ```bash git clone https://github.com/BerriAI/litellm.git cd litellm make install-dev # Install development dependencies make format # Format your code make lint # Run all linting checks make test-unit # Run unit tests make format-check # Check formatting only ``` For detailed contributing guidelines, see [CONTRIBUTING.md](CONTRIBUTING.md). > **📖 Contributing to documentation?** The LiteLLM docs have moved to a separate repository: [BerriAI/litellm-docs](https://github.com/BerriAI/litellm-docs). Please open doc PRs there. Docs are served at [docs.litellm.ai](https://docs.litellm.ai). ## Code Quality / Linting LiteLLM follows the [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html). Our automated checks include: - **Black** for code formatting - **Ruff** for linting and code quality - **MyPy** for type checking - **Circular import detection** - **Import safety checks** All these checks must pass before your PR can be merged. # Support / talk with founders - [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version) - [Community Discord 💭](https://discord.gg/wuPM9dRgDw) - [Community Slack 💭](https://www.litellm.ai/support) - Our emails ✉️ ishaan@berri.ai / krrish@berri.ai # Contributors