# Introduction

High-performance LLM API gateway in Rust — one API format, many providers, sub-millisecond overhead.

CrabLLM is a high-performance LLM API gateway written in Rust. It sits between your application and LLM providers, exposing an OpenAI-compatible API surface.

One API format. Many providers. Low overhead.

## What it does

You send requests in OpenAI format to CrabLLM. It routes them to the configured provider — OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock, or Ollama — translating the request and response as needed.

Your application talks to one endpoint. CrabLLM handles the rest:

- **Provider translation** — Anthropic, Google, and Bedrock have their own API formats. CrabLLM translates automatically.
- **Routing** — Weighted random selection across multiple providers for the same model. Automatic fallback when a provider fails.
- **Streaming** — SSE streaming proxied without buffering.
- **Auth** — Virtual API keys with per-key model access control.
- **Extensions** — Rate limiting, caching, cost tracking, budget enforcement.
- **Management** — Dynamic provider and key management via admin API and crabctl CLI. OpenAPI docs at `/docs`.

## Why Rust

- **Sub-millisecond overhead** — no GC pauses, no interpreter startup.
- **Memory safety** — without runtime cost.
- **Concurrency** — Tokio async runtime handles thousands of concurrent streaming connections efficiently.
- **Deployment** — single static binary. No interpreter, no virtualenv, no Docker required.

## Feature comparison

| Feature | LiteLLM | CrabLLM |
| --- | --- | --- |
| `/chat/completions` | yes | yes |
| `/embeddings` | yes | yes |
| `/models` | yes | yes |
| `/v1/messages` (Anthropic native) | — | yes |
| OpenAI provider | yes | yes |
| Anthropic provider | yes | yes |
| Google Gemini provider | yes | yes |
| Azure OpenAI provider | yes | yes |
| AWS Bedrock provider | yes | yes |
| Tool/function calling | yes | yes |
| SSE streaming | yes | yes |
| Virtual keys + auth | yes | yes |
| Weighted routing | yes | yes |
| Model aliasing | yes | yes |
| Retry + fallback | yes | yes |
| Rate limiting (RPM/TPM) | yes | yes |
| Cost/usage tracking | yes | yes |
| Budget enforcement | yes | yes |
| Request caching | yes | yes |
| Image/audio endpoints | yes | yes |
| Admin CLI (crabctl) | — | yes |
| Dynamic provider management | — | yes |
| OpenAPI / Scalar docs | — | yes |
| Storage (memory) | yes | yes |
| Storage (persistent) | Postgres | SQLite |
| Redis storage | yes | yes |

Getting Started

Install CrabLLM, configure providers, and send your first request in under a minute.