# Architecture

Design principles, workspace layout, and request flow through the CrabLLM gateway.

## [Principles](/content/docs/crabllm/architecture#principles/index.html)

- **Simplicity over abstraction.** No trait where a function suffices.
- **Single responsibility.** Each crate has one focused job.
- **OpenAI as canonical format.** Providers translate to/from it.
- **Streaming first-class.** Never buffer a full response when streaming.
- **Configuration-driven.** Provider setup and routing from config, not code.
- **Minimal gateway latency.** Avoid hot-path allocations.

## [Workspace layout](/content/docs/crabllm/architecture#workspace-layout/index.html)

```
crabllm/
  crates/
    crabllm/    — binary (serve, init, openapi subcommands)
    crabctl/    — admin CLI for managing a running gateway
    core/       — shared types, config, errors
    provider/   — provider enum + translation modules
    proxy/      — HTTP server, routing, extensions, admin API
    mlx/        — Apple Silicon local inference via MLX
    llamacpp/   — cross-platform local inference via llama.cpp
    bench/      — benchmark mock backend
```

## [Crates](/content/docs/crabllm/architecture#crates/index.html)

### [crabllm](/content/docs/crabllm/architecture#crabllm/index.html)

Binary entry point. Three subcommands:

- **`serve`** (default) — loads TOML config, builds the provider registry, initializes storage + extensions, starts the Axum HTTP server. Flags: `--config`, `--bind`, `-v/-vv/-vvv` for verbosity.
- **`init`** — generates a starter `crabllm.toml` in the current directory.
- **`openapi`** — dumps the OpenAPI spec as JSON or a self-contained Scalar HTML page.

### [crabctl](/content/docs/crabllm/architecture#crabctl/index.html)

Admin CLI for managing a running gateway over HTTP. Supports key management (`keys list|create|get|update|delete`), provider management (`providers list|create|get|update|delete`), usage/budget/logs queries, and cache clearing. See [Management](/content/docs/crabllm/features/management/index.html).

### [core](/content/docs/crabllm/architecture#core/index.html)

Shared types with no business logic. Contains:

- **Config** — `GatewayConfig` with env var interpolation.
- **Types** — OpenAI-compatible wire format structs (request, response, chunk).
- **Provider trait** — async trait with methods for chat, streaming, embeddings, images, audio. Uses RPITIT for zero-cost dispatch.
- **Error** — error enum with transient detection for retry logic.
- **Storage** — async KV trait with memory, SQLite, and Redis backends.
- **Extension** — hook trait for the request pipeline.

### [provider](/content/docs/crabllm/architecture#provider/index.html)

Provider dispatch. `ProviderRegistry` maps model names to weighted deployment lists. Supports alias resolution, weighted random selection, and per-model provider lookup. Generic over `P: Provider` so it unifies remote APIs, MLX, and llama.cpp.

### [proxy](/content/docs/crabllm/architecture#proxy/index.html)

Axum HTTP server. Route handlers implement retry + fallback across deployments. Auth middleware validates virtual keys. Five built-in extensions run as in-handler hooks. Admin API routes at `/v1/admin/*` for dynamic key and provider management. OpenAPI/Scalar docs at `/docs` and `/openapi.json` when enabled.

### [mlx](/content/docs/crabllm/architecture#mlx/index.html)

Local inference on Apple Silicon. Thin Rust wrapper around a Swift static library using the MLX framework. Multi-model cache with idle eviction. Supports chat completions (streaming + non-streaming) with tool calling. macOS and iOS only — stubs out on other platforms.

### [llamacpp](/content/docs/crabllm/architecture#llamacpp/index.html)

Cross-platform local inference. Manages the lifecycle of spawned `llama-server` processes — auto-downloads the binary, pulls models from the Ollama registry, spawns per-model servers on demand, and evicts idle servers.

## [Request flow](/content/docs/crabllm/architecture#request-flow/index.html)

ProviderExtensionsHandlerAuthClientProviderExtensionsHandlerAuthClientOpenAI-format requestValidate bearer tokenResolve model (aliases)on_request (rate limit, budget)on_cache_lookupDispatch (weighted selection)Translate, call upstream, translate backon_response / on_chunkResponse

### Management

Dynamic provider and key management via crabctl CLI, admin API, and OpenAPI docs.

### Libraries

Standalone Rust crates for local inference — MLX (Apple Silicon) and llama.cpp (cross-platform).