Skip to main content

Rust Migration

LiteLLM is moving to Rust

See how much of the move to Rust is built and what is next.

Migration tracker

Where we are today

22% of built, shown as a .

22 percent of Major APIs is built in v1.106.0-rc.2. /messages e2e, simple is planned to reach 47 percent. /messages e2e, all is planned to reach 66 percent. The goal targets 100 percent by 2026-12-31. Horizontal distance represents elapsed time on the same scale for releases, plans, and the goal.OctNovDec0%50%100%1.102 rc1Sep 131.104 rc1Sep 2822%1.106 rc2Oct 13 · latest47%/messagese2e, simpleNov 766%/messagese2e, allNov 21Major APIs: 100% by 2026-12-31100% goalGoalDec 31

Result

The result of the move to Rust

This is where the migration takes us. AIGatewayBench measured four gateways against the same mock server: the Rust gateway, Portkey, Bifrost, and the current Python proxy. The test measured the delay and the memory that each gateway adds.

Added latency at p99

  • 0.7 msLiteLLM Rust
  • 2.3 msPortkey
  • 4.5 msBifrost
  • 257.7 msLiteLLM Python v1

Logarithmic scale. A low value is better.

Maximum memory use

  • 21.8 MBLiteLLM Rust
  • 90.4 MBPortkey
  • 199.1 MBBifrost
  • 329.5 MBLiteLLM Python v1

A low value is better.

These numbers are the delay the gateway itself adds. Each gateway handled 5000 requests for each endpoint, and p99 is the slowest request in every 100. The AIGatewayBench report gives the full method, the raw data, and the limits of the test.

Engineering updates

How we are getting there

  1. · Blog post

    OCR uses Rust by default starting with v1.102.0-rc.1
  2. · Blog post

    Benchmarking the LiteLLM Rust AI Gateway: Overhead, Memory, and Cost
  3. · Blog post

    Migrating LiteLLM to Rust - Building the Fastest and Litest AI Gateway

FAQ

Frequently asked questions

  • How is the percentage calculated?

    Every piece of work has a size in rough engineer-weeks, and a component's points are the sizes of its work. A piece earns 80% of its size once it is in a release but incomplete or not yet trusted, and all of it once it is done. Major APIs is 146 points in all.

    Major APIsPointsTodayBy Nov 21
    Deployment10010
    Rust core888
    Messages translation72.47
    Chat completions translation130.60.6
    Responses translation6.51.61.6
    Python-only request features38.50.815.6
    DB access808
    Auth and secrets138.612.6
    Budgets and rate limits504
    Spend tracking313
    Router806.4
    Logging and callbacks123.29.6
    Response cache126.69
    Config201.6
    Total14632.897
  • What do Partial and Done mean?

    A level says how confident we are in the work itself, not whether anything uses it yet. Groundwork that is finished and trusted but waits for the gateway to deploy is already done.

    • Partial

      In a release, but incomplete or not yet trusted, so it usually ships opt-in.

    • Done

      Complete and trusted, whether or not it runs by default or anything uses it yet.