Rust Migration
LiteLLM is moving to Rust
See how much of the move to Rust is built and what is next.
Migration tracker
Where we are today
22% of built, shown as a .
Deployment0%of 10 pts
- Gateway binary in the Docker image
- Forward unserved requests to Python
- Dispatch crate and gap table
- Per model group plans at config load
- Identify, dispatch and admit layers
Rust core100%of 8 pts
- Python bridge
- Python host calls
- HTTP client
- Streaming
- Async bridge
- Tracing
- Model catalog
- Config parsing
Messages translation34%of 7 pts
- Anthropic
- DeepSeek
- Vertex AI
- Bedrock
- Bedrock Mantle
- Azure AI
- MiniMax
- GitHub Copilot
- Tencent
- Eden AI
- OpenAI-compatible
Chat completions translation5%of 13 pts
- OpenAI
- Azure OpenAI
- Anthropic
- Bedrock
- Vertex AI
- Gemini
- Azure AI
- Mistral
- Cohere
- Groq
- xAI
- DeepSeek
- Together AI
- Fireworks AI
- OpenRouter
- Databricks
- watsonx
- Perplexity
- DeepInfra
- Cerebras
- NVIDIA NIM
- SageMaker
- Hugging Face
- Replicate
- AI21
- SambaNova
- Nebius
- Novita AI
- Hyperbolic
- Baseten
- Moonshot AI
- DashScope
- Volcengine
- MiniMax
- GitHub Copilot
- Snowflake
- Oracle OCI
- Vercel AI Gateway
- LM Studio
- Ollama
- vLLM
- OpenAI-compatible
Responses translation25%of 6.5 pts
- OpenAI
- Azure OpenAI
- Azure AI
- Bedrock
- Bedrock Mantle
- xAI
- OpenRouter
- Databricks
- Fireworks AI
- Perplexity
- GitHub Copilot
- ChatGPT
- vLLM
- Volcengine
- Eden AI
- Manus
- Sail
- LiteLLM Proxy
- OpenAI-compatible
Python-only request features2%of 38.5 pts
- Mock responses
- Proxy MCP tools
- Caller HTTP clients
- Request, stream and phase timeouts
- Agentic loop hooks
- TLS security level and curves
- modify_params rewrites
- Prompt cache breakpoint injection
- Anthropic workload identity tokens
- Thinking signature retry
- Cost in streamed usage
- Web search interception
- Messages interceptors
- Safeguards classifier
- drop_params
- Extra body fields
- Prompt management
- Model param gating
- Chat completions streaming
- Per-call model_list
- model_alias_map
- text_format parsing
- Responses through Chat completions
- Deployment and encoded item ids
- Responses input rewrites
- Provider affinity header
- Managed file and vector store ids
- Cache marks, tool search and tool schema repair
- Forwarded client headers
- Sync deployment hook
DB access0%of 8 pts
- Query registry and Python codegen
- Python runtime and PrismaExecutor
- Schema and codegen checks in CI
- Key, end user and spend queries
Auth and secrets66%of 13 pts
- AWS auth
- Azure auth
- Google Cloud auth
- Secret managers
- Virtual key lookup and cache
- Key cache invalidation across hosts
- Shared authorization crate
- JWT auth
Budgets and rate limits0%of 5 pts
- Shared rate limit scripts and key naming
- Budget checks
Spend tracking33%of 3 pts
- Cost calculation
- Spend writes through the shared Redis buffer
Router0%of 8 pts
- Shared routing strategy crate
- Cooldown and health state
- Retries and fallbacks
- Models stored in DB
Logging and callbacks27%of 12 pts
- Logging callbacks
- Shared logging payload
- Prometheus
- S3 and other exporters
- Hook pipeline contract
- Custom Python callbacks forwarded to Python
Response cache55%of 12 pts
- In-memory and Redis backends
- Python compatibility paths removed
- Cache built from config in the gateway
- Cache gaps in dispatch
- Cache hits in spend and logs
- Chat and Responses caching
- Other storage backends
- Cache management endpoints
Config0%of 2 pts
- Config stored in DB
Result
The result of the move to Rust
This is where the migration takes us. AIGatewayBench measured four gateways against the same mock server: the Rust gateway, Portkey, Bifrost, and the current Python proxy. The test measured the delay and the memory that each gateway adds.
Added latency at p99
- 0.7 msLiteLLM Rust
- 2.3 msPortkey
- 4.5 msBifrost
- 257.7 msLiteLLM Python v1
Logarithmic scale. A low value is better.
Maximum memory use
- 21.8 MBLiteLLM Rust
- 90.4 MBPortkey
- 199.1 MBBifrost
- 329.5 MBLiteLLM Python v1
A low value is better.
These numbers are the delay the gateway itself adds. Each gateway handled 5000 requests for each endpoint, and p99 is the slowest request in every 100. The AIGatewayBench report gives the full method, the raw data, and the limits of the test.
Engineering updates
How we are getting there
FAQ
Frequently asked questions
How is the percentage calculated?
Every piece of work has a size in rough engineer-weeks, and a component's points are the sizes of its work. A piece earns 80% of its size once it is in a release but incomplete or not yet trusted, and all of it once it is done. Major APIs is 146 points in all.
Major APIs Points Today By Nov 21 Deployment 10 0 10 Rust core 8 8 8 Messages translation 7 2.4 7 Chat completions translation 13 0.6 0.6 Responses translation 6.5 1.6 1.6 Python-only request features 38.5 0.8 15.6 DB access 8 0 8 Auth and secrets 13 8.6 12.6 Budgets and rate limits 5 0 4 Spend tracking 3 1 3 Router 8 0 6.4 Logging and callbacks 12 3.2 9.6 Response cache 12 6.6 9 Config 2 0 1.6 Total 146 32.8 97 What do Partial and Done mean?
A level says how confident we are in the work itself, not whether anything uses it yet. Groundwork that is finished and trusted but waits for the gateway to deploy is already done.
Partial
In a release, but incomplete or not yet trusted, so it usually ships opt-in.
Done
Complete and trusted, whether or not it runs by default or anything uses it yet.