Deployed models forget. They run to a knowledge cutoff, and the alternative, retraining on new data, tends to wipe out what they already knew in a failure called catastrophic forgetting. Leaning on a context window helps only until the session ends.
Researchers at Google, Cornell and the University of Southern California borrowed a biological answer. Their sleep framework alternates an active waking phase, where the model takes in new material and parks it in fast temporary memory, with an offline phase where it consolidates. Borrowing the vocabulary of neuroscience does not change the mechanism, which runs entirely inside a fixed architecture under human-defined training rules.
Consolidation works in stages. The model unlocks pre-allocated mixture-of-experts memory blocks and fits low-rank experts into stable modules, then transfers short-term knowledge into those long-term weights using on-policy distillation and an imitation reward based on semantic similarity. Fragile temporary parameters get pruned afterward.
A dreaming stage follows. The router randomly activates non-standard experts, generating synthetic practice samples that mix disparate domains, filters them by gradient-based selection, and refines the survivors.
The results are unusual. On the BABILong long-context benchmark the method scaled close to perfectly to 10M tokens, ahead of GPT-4 and Titans. In continual translation of unseen languages such as Manchu and Kalamang, performance barely degraded across stages where in-context learning and supervised fine-tuning collapsed. On AIME 24, AIME 25 and HMMT 25 the sleep-enhanced base model hit 79.2, 69.0 and 46.1 percent.
