arxiv:2604.01591

ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement

Published on Apr 2

· Submitted by

Difan Jiao on Apr 8

University of Toronto CSSLab

Upvote

Authors:

Difan Jiao ,

Qianfeng Wen ,

Zhenwei Tang ,

Abstract

ThinkTwice is a two-phase framework that jointly optimizes large language models for reasoning and self-refinement using Group Relative Policy Optimization, demonstrating improved performance on mathematical reasoning benchmarks.

AI-generated summary

We introduce ThinkTwice, a simple two-phase framework that jointly optimizes LLMs to solve reasoning problems and refine the answers, based on Group Relative Policy Optimization (GRPO). In each pair of training steps, ThinkTwice first optimizes the model on solving reasoning problems, then optimizes it on refining its own solutions to the same problems, using the same binary correctness reward in both phases without correctness signals or critique annotations. Across five mathematical reasoning benchmarks and two model families including Qwen3-4B and Olmo3-7B, ThinkTwice substantially improves both reasoning and refinement performance over competitive online policy optimization baselines. Specifically, on Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step, measured by pass@4. Analysis of the training dynamics of ThinkTwice reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training and naturally shifts toward preserving already-correct solutions as the model improves, yielding a more rectified reward signal. Our work establishes joint training of reasoning and self-refinement as a principled and effective methodology for RLVR.

View arXiv page View PDF GitHub 10 Add to collection

Community

difanjiao

Paper author Paper submitter 1 day ago

loveblairsky

1 day ago

👍

mishig

about 19 hours ago

ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinement

ThinkTwice is a two-phase GRPO framework that trains large language models to both solve reasoning problems and refine their own solutions, using only binary correctness rewards and no critique annotations. The method discovers an implicit "rectify-then-fortify" curriculum where early training corrects errors and later training preserves already-correct solutions.

Key Idea

The core insight is that reasoning and self-refinement can be jointly optimized in a simple two-phase loop. In Phase 1 the model generates an initial solution to a reasoning problem. In Phase 2 the model attempts to refine that solution. Both phases use the same binary correctness reward signal under GRPO, eliminating the need for expensive critique or preference annotations.

Method / Approach

During training, an emergent "rectify-then-fortify" curriculum naturally arises. In early iterations the refinement phase primarily corrects wrong initial answers, learning to identify and fix mistakes. As training progresses and the model's initial solutions improve, the refinement phase shifts to preserving correct answers — learning when not to change things. This two-regime dynamic is not engineered but emerges from the joint optimization.

Results

On Qwen3-4B, ThinkTwice outperforms standard GRPO on AIME by 5 percentage points before refinement and 11.5 percentage points after refinement. The approach generalizes across architectures, showing consistent gains on both Qwen3-4B and OLMo3-7B.