CMU Releases CompressionBench: Six Coding Agents Get 8 Hours Each to Compress LLMs, 29 of 36 Models Pass, and Answer Tests Catch Failures Perplexity Misses

Compression as a terminal task

Nine authors from Carnegie Mellon University's Infini-AI-Lab, including Beidi Chen, submitted CompressionBench to arXiv on November 4 and released the code and tasks. The question is simple: can a coding agent shrink a large model to a given size on its own and keep it useful? Each task gives the agent the source weights, a storage limit, one H200 NVL, a pinned vLLM version and an eight-hour budget. The agent decides how to combine quantization, mixed precision, pruning, low-rank representations and distillation, and must deliver compressed weights plus the inference code that runs them. The limit applies both to the saved package and to the weights resident in GPU memory during inference, closing the loophole of a small file that expands back to full precision at run time.

The agents were Claude Sonnet 5.5, Claude Opus 5.5, GLM 5.3, Kimi K3.1 Pro, MiniMax M3.1 Flash and DeepSeek V4.1 Flash 1017, all run through Claude Code at maximum reasoning effort. The source models were two MoE checkpoints, Nemotron-3.5 Super VL 120B-A12B and Qwen3.5 122B-A10B (32 GiB and 48 GiB limits), and the dense Qwen3.8 27B (8 GiB and 12 GiB).

Results: all pass at looser limits, some fall short at tighter ones

During search the agents saw only perplexity feedback. After the deadline a separate environment verified each model, requiring mean accuracy on GSM8K, HumanEval and TriviaQA (50 questions each) to retain 80% of BF16. Of 36 submissions, 29 passed, including all 18 at the looser limits; at the tighter limits, six were rejected for accuracy and one timed out while generating answers. Taking the lowest-perplexity model for each task gave an average 3.4% perplexity increase over BF16 and 77.33% mean accuracy, versus 79.56% for BF16. By perplexity, the best agent-built model in each task beat each of the five released quantizations measured, while a uniform-bit-width GPTQ baseline passed only on the looser MoE tasks. Sonnet had the best average perplexity rank and Opus came second; the largest API spend did not produce the best rank.

What perplexity alone misses

The more useful part of the paper analyzes the agents' recorded searches. In 24 revisions that left package size almost unchanged, 21 lowered perplexity by improving the quantizer or redistributing precision, and complete-model feedback reversed 17 of 79 earlier local rankings. But perplexity can mislead. In one dense-model run, Opus kept improving perplexity while GSM8K accuracy fell from 88% to 72%, and pruned models from GLM and MiniMax produced repeated or truncated HumanEval completions. In the other direction, MiniMax compared token probabilities under batched and per-expert inference, found an expert-indexing bug, and fixing it cut the perplexity increase over BF16 from about 11% to 4.5%.

The authors name representation design, experimental judgment and capability validation as the keys to agent-driven compression. Each agent-task pair was run only once, so the findings still need repeated experiments; for teams using agents to optimize models, the paper's message is that acceptance has to include real generation tests, not just perplexity.