Test-Time Scaling Makes Overtraining Compute-Optimal

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

• To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining (2026)
• Prescriptive Scaling Reveals the Evolution of Language Model Capabilities (2026)
• Configuration-to-Performance Scaling Law with Neural Ansatz (2026)
• Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization (2026)
• Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning (2026)
• Optimal Splitting of Language Models from Mixtures to Specialized Domains (2026)
• The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data (2026)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Interesting breakdown of this paper on arXivLens: https://arxivlens.com/PaperView/Details/test-time-scaling-makes-overtraining-compute-optimal-6527-8b212681
Covers the executive summary, detailed methodology, and practical applications.

Here are the main results from the paper, which introduces Train-to-Test (T²) scaling laws that jointly optimize pretraining and test-time compute:

Core Finding: Overtraining Becomes Compute-Optimal

The central result is that when you account for inference costs (test-time scaling via repeated sampling), the optimal pretraining strategy shifts dramatically toward smaller, heavily overtrained models—far beyond the standard Chinchilla recommendation of ~20 tokens per parameter.

Figure 2: Both T² approaches forecast that optimal pretraining requires extreme overtraining (left panel) with dramatically smaller model sizes (middle panel) compared to Chinchilla scaling, when accounting for a fixed inference budget (C_inf ≈ 140B FLOPs, roughly equivalent to one forward pass of a 70B model).

Key Quantitative Shifts

• Tokens per parameter: While Chinchilla recommends ~20 tokens/parameter, T² scaling suggests ratios orders of magnitude higher (the exact multiplier depends on the inference budget, but the trend shows extreme overtraining)
• Model size: Optimal models are substantially smaller than Chinchilla would prescribe for the same training compute
• Inference allocation: Smaller models get allocated more samples (k) since k = C_inf/(2N), creating a multiplicative trade-off

Validation: T² Extrapolates to Real Overtrained Checkpoints

The authors validate their scaling laws by training 21 new heavily overtrained checkpoints (beyond the standard Chinchilla suite) and confirming the predictions hold:

Figure 4: Both T² approaches successfully extrapolate to overtrained checkpoints. Approach 1 achieves 2.8% relative error, while Approach 2 achieves 8.4%.

Table 1 results (selected examples from the paper):

All comparisons use identical inference budgets (C_inf = 2×10⁹ FLOPs), meaning the smaller overtrained models use more repeated samples (k) than larger Chinchilla-optimal models.

Post-Training Robustness

The findings persist after supervised fine-tuning (SFT) and standard fine-tuning (FT), though the overtraining recommendation is slightly subdued (consistent with prior work showing overtrained models are harder to fine-tune):

Figure 5: After post-training on ARC-Easy, SciQ, and OpenBookQA, the optimal frontier still favors smaller overtrained models over Chinchilla-optimal baselines, confirming that T² scaling remains relevant for deployed models.

The T² Scaling Approaches

The paper proposes two complementary modeling approaches that agree on the overtraining conclusion:

1. Approach 1 (Blue): Extends Chinchilla loss scaling with a power-law term for repeated sampling:

   • L̂(N,D,k) = E + A/N^α + B/D^β + G/k^γ

2. Approach 2 (Red): Models pass@k directly using Beta regression on per-question accuracies, accounting for the concavity of the pass@k metric (Jensen's gap)

Both approaches show that under inference-corrected budgets, Chinchilla scaling exhibits non-monotonic improvement (larger training budgets can hurt performance if model size is wrong), while T² scaling shows consistent monotonic gains when jointly optimizing pretraining and test-time compute.

Practical Recommendation

If you know your test-time scaling budget (i.e., you'll use repeated sampling with pass@k evaluation), you should:

1. Train a smaller model than Chinchilla would suggest
2. Train it for many more tokens (heavy overtraining)
3. Use more inference samples at test time (which is cheaper per-sample with the smaller model)

This "small but overtrained" configuration maximizes end-to-end performance under joint train-test compute budgets.