qMLX icon qMLX

Disk KV restore for hybrid MoE models on Apple Silicon. A follow-up on a 130,000-token conversation drops from a multi-minute cold prefill to a sub-second restore.

uv add qmlx-serve
View on GitHub Read the docs
Qwen3.5-122B-A10B M3 Ultra OpenAI + Anthropic API MIT + Apache-2.0
Cold prefill versus warm disk restore on a repeated 32k prompt
Repeated 32k prompt on an M3 Ultra: 88s cold prefill against a 0.64s warm restore.

The cache is the whole game.

Qwen3.5-122B is hybrid: roughly 75% of its layers are recurrent DeltaNet, which cannot be rewound, so the standard in-memory prefix cache misses 100% of the time. Disk KV restore is not a fallback here, it is the entire cache.

137× faster on a warm restore than a cold prefill (repeated 32k prompt)
88s → 0.64s time to first token on that prompt, cold against warm
0 / 109 in-memory hits against disk hits in a normal window, which is why disk carries it

Serve it.

Drop-in OpenAI and Anthropic APIs, the same surface as upstream Rapid-MLX. One flag turns the disk cache on.

rapid-mlx serve mlx-community/Qwen3.5-122B-A10B-4bit \
  --text-only --host 0.0.0.0 --port 8095 \
  --max-num-seqs 1 \
  --enable-prefix-cache --prefix-cache-index radix \
  --enable-disk-kv-restore \
  --kv-disk-checkpoint-interval 256

# next turn on a 130k conversation:
# sub-second restore instead of a cold prefill

Why --text-only

The vision path is incompatible with the hybrid continuous-batching the cache work depends on, so text serving is required for now.

The checkpoint the next turn needs never gets evicted by unmatchable interval writes under the disk cap. That was the bug that made the cache look broken on real agentic traffic.

Metrics are phase-split and honest: real decode tok/s over the decode window, real prefill throughput excluding cached tokens, disk-restore hit rate, and TTFT.

What is in it.

Disk KV checkpoint and restore

Checkpoints the attention KV to SSD and pages it back on the next turn, for hybrid recurrent plus attention MoE caches. Int4 checkpoints, dequantised layer by layer on restore.

Matchable-aware eviction

The checkpoint the next turn actually needs never gets evicted by unmatchable interval writes hitting the disk cap. This is the fix that made the cache hold on agentic-coding traffic.

Honest, phase-split metrics

Real decode tok/s over the decode window, real prefill throughput excluding cached tokens, disk-restore hit rate, TTFT. No amortised (prompt+gen)/wall throughput number.

Live divergence logging

Pinpoints the exact token where a prefix-cache match broke, so this class of bug is diagnosable in minutes instead of a day of guessing.

Qwen-first, and honest about it

Runs Qwen3.5-122B-A10B on an M3 Ultra with 96GB of unified memory. Decode slows gradually with context but there is no cliff, so it stays usable well past 100k tokens. Windowed attention to flatten that curve further is on the roadmap.

Drop-in API

OpenAI and Anthropic compatible endpoints, the same serving surface as upstream Rapid-MLX. Point an existing client at it and go.