Progressive Cramming

Cramming compresses a span of text into a single learnable memory embedding: we freeze a pretrained LM and optimise one input vector $\mathbf{e}$ by gradient descent until the frozen model reconstructs the original tokens from it. Progressive cramming grows the target span one token at a time, advancing only after the current span reconstructs exactly — yielding an honest, per-sample compression horizon with 100% greedy reconstruction. We find horizons of hundreds-to-thousands of tokens across model families, that the optimisation trajectories are intrinsically low-dimensional, and a sharp caveat: reconstruction ≠ understanding — prepending a perfectly crammed embedding drops downstream accuracy and collapses generative MMLU to ~0%.

The method

A frozen LM and one trainable input embedding, optimised per text — grown one token at a time.

Progressive cramming --progressive_train

Grow the target span one token at a time, warm-starting the embedding between stages and advancing only on exact reconstruction. Halting at the first failure gives a sharp, honest per-sample capacity boundary — the compression horizon — with 100% greedy reconstruction (unlike full cramming, whose ~99% teacher-forced accuracy still derails under greedy decoding).

Progressive cramming: a frozen LM, a trainable embedding warm-started between stages, target grown one token at a time. — **Progressive cramming.** The frozen model reconstructs the prefix from the trainable embedding; the target is extended one token per stage and the embedding is warm-started from the previous stage, tracing an optimiser trajectory.

How much fits in one embedding

Maximum prefix a frozen model reconstructs exactly from a single embedding via progressive cramming — and the boost from optimising inside a low-dimensional subspace.

Model	Compressed tokens	+ Low-dim projection	Boost
Llama-3.1-8B	1438	1697	1.2×
Pythia-1.4B	430	500	1.2×
SmolLM2-1.7B	335	957	2.9×
Gemma-3-4B	214	697	3.3×

Tokens = longest prefix reconstructed exactly (mean over 50 PG-19 samples). Low-dim uses $k=256$ (Gemma $k=32$). These are the runs behind the paper's progressive_modifications table — step through them in the explorer below, or download the full per-stage trajectories from Hugging Face.

Optimisation trajectory — Llama-3.1-8B. The per-sample embedding projected onto its top PCA components (PC1–PC2). Black dots: the optimised embedding at each prefix length. Coloured regions: the local accuracy landscape at successive lengths; saturation $\propto$ reconstruction accuracy. Trajectories are intrinsically low-dimensional despite living in $\mathbb{R}^{4096}$, and the high-accuracy basin shrinks as the sequence grows.

Low-dim projection — Llama-3.1-8B. The same family with $\mathbf{e}=W\mathbf{z}+\mathbf{b}$ ($k=256$), zoomed on the progressive frontier. Coloured regions: the >90% accuracy basin at every 100th stage; its area contracts monotonically with prefix length. The low-rank subspace lets the optimiser escape local minima and explore multiple basins — reaching far longer horizons.

Trajectories are low-dimensional

30–100 PCA components capture 99% of the variance of an entire cramming trajectory, and the count grows only ~logarithmically with sequence length. The same warm-started optimisation also leaves a per-token fingerprint: harder (higher-surprisal) tokens take more optimiser steps to absorb — exactly what the explorer visualises.

Reconstruction ≠ understanding

A perfectly crammed embedding stores enough to regenerate the text — but almost nothing the model can use. Prepending it drops benchmark accuracy, even with the original prefix still in context.

Model	HellaSwag base	HellaSwag + emb	ARC base	ARC + emb
Pythia-1.4B	44.0%	37.6%	53.1%	49.1%
SmolLM2-1.7B	53.6%	37.0%	67.3%	56.0%
Llama-3.1-8B	61.9%	40.8%	63.5%	43.3%

Likelihood evaluation: a candidate suffix is scored by NLL given [crammed embedding · prefix · suffix]. On generative 5-shot MMLU the effect is starker — accuracy collapses to ~0% with the optimised embedding, while a random-vector control stays at baseline. The information is present but acts as brittle steering, not transferable semantics. Attention-knockout analysis traces both reconstruction and capability collapse to the model's early layers.

Interactive per-token explorer

Step through a real cramming run token by token. Drag the horizon slider to watch the frozen model absorb the prefix one token at a time; hover any token for its surprisal and the optimiser steps it cost. Notice how high-surprisal tokens spike the step cost — and where the horizon hits a wall.

Model

Method

Sample

PG-19 · greedy reconstruction is exact up to the horizon

Horizon position

steps to converge per token current horizon wall (first token beyond horizon)

Loading trajectories…

Surprisal $s(L)=-\log_2 p(x_L\mid x_{\lt L})$ is the frozen base model's per-token difficulty (bits); steps-to-converge is the marginal optimiser cost of absorbing that token. Data is exported offline from the progressive_modifications runs (one frozen forward pass per sample for surprisal; steps read from each run's per-stage dataset) — the full per-stage trajectories are published on Hugging Face.

✓

Citation

BibTeX

@inproceedings{tarasov2026progressive,
  title     = {Progressive Cramming: Reliable Token Compression and What It Reveals},
  author    = {Tarasov, Dmitrii and Lashukov, Timofei and Goncharova, Elena and Kuznetsov, Andrey},
  booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},
  year      = {2026}
}

Reference implementation, quickstart, and full reproduction recipe: github.com/FusionBrainLab/progressive_cramming. Builds on the cramming task of Kuratov et al. (2025).