Frontier AI on a laptop moved a step closer with FreeToken, an open-source inference engine from UC Berkeley and MIT. The project aims to run massive mixture-of-experts models on consumer hardware, and its authors read like a who’s who of systems research: Databricks co-founders Matei Zaharia and Ion Stoica, plus Song Han and Kurt Keutzer.
Sparse MoE models only activate part of their parameters per token, but decoding still touches hundreds of billions of inactive weights. Data centers hide that traffic behind NVLink; consumer PCIe links and host RAM turn it into a severe bottleneck, and existing edge runtimes stall the GPU every time they stream a missing expert in from system memory.
The engine swaps rigid offloading for a live balancing act. A scheduling policy called q* splits each token’s work between CPU cores and GPU tensor cores, reading real-time interconnect throughput to decide where computation happens. A fast weight format and double buffering let weight streaming overlap with active layers, and an elastic memory manager reshuffles VRAM between KV cache and resident experts on the fly. Semantic anchor checkpointing lets agents reuse cached states when prompts mutate.
Results from the paper include Qwen3.6-35B running at roughly 39 tokens per second on an 8GB RTX 4060 laptop, DeepSeek-V4-Flash at 284B parameters on an RTX 5090 desktop, and GLM-5.2 at 753B on a single workstation GPU. Against Ollama and llama.cpp, decode speeds are 3 to 4 times faster and prefill 6 to 30 times faster on equivalent MoE models.
