Post
278
# Three Teachers, One Student: Dual-Cognition Reasoning at 1.7B
We distilled Qwen3-30B-A3B into 1.7B students that critique their own reasoning. H100, BF16, Apache 2.0. Here's our pipeline.
**Stage 1 — Three Teachers, Three Profiles.** Same 30B base, three variants: Instruct (structured output), Thinking (extended deliberation), Coder (STEM decomposition). Each distillation uses proof-weighted KD — 2.25× amplified loss on reasoning tokens, decaying to 1.1×. The student learns *where to think harder*, not just what to output.
**Stage 2 — Topology-Aware KD (TKD).** Standard KD treats the teacher's distribution as smooth. Language isn't smooth — it has topic shifts, reasoning pivots, register changes. We use Discrepancy Calculus to detect these structural boundaries, then amplify loss at jumps (3σ threshold) and cut training windows at low-discrepancy positions. The student preserves the teacher's structural knowledge, not just surface statistics.
**Stage 3 — Ghost Imprinting.** Sequential distillation from different teachers leaves residual fields in weight space that neither teacher put there individually. The Cantor component of BV decomposition, applied to parameters. Models distilled Thinking→Coder exhibit deliberation patterns from the Thinking teacher that survived Coder overwriting. Emergent capability from structural residuals.
**Stage 4 — DualMind.** One model, two voices, shared weights:
The multi-model collision array collapsed into a single architecture. Role tokens, no extra parameters.
For the full method:
reaperdoesntknow/DualMind_Methodolgy
doi:10.57967/hf/8184.
We distilled Qwen3-30B-A3B into 1.7B students that critique their own reasoning. H100, BF16, Apache 2.0. Here's our pipeline.
**Stage 1 — Three Teachers, Three Profiles.** Same 30B base, three variants: Instruct (structured output), Thinking (extended deliberation), Coder (STEM decomposition). Each distillation uses proof-weighted KD — 2.25× amplified loss on reasoning tokens, decaying to 1.1×. The student learns *where to think harder*, not just what to output.
**Stage 2 — Topology-Aware KD (TKD).** Standard KD treats the teacher's distribution as smooth. Language isn't smooth — it has topic shifts, reasoning pivots, register changes. We use Discrepancy Calculus to detect these structural boundaries, then amplify loss at jumps (3σ threshold) and cut training windows at low-discrepancy positions. The student preserves the teacher's structural knowledge, not just surface statistics.
**Stage 3 — Ghost Imprinting.** Sequential distillation from different teachers leaves residual fields in weight space that neither teacher put there individually. The Cantor component of BV decomposition, applied to parameters. Models distilled Thinking→Coder exhibit deliberation patterns from the Thinking teacher that survived Coder overwriting. Emergent capability from structural residuals.
**Stage 4 — DualMind.** One model, two voices, shared weights:
<explore> — free derivation, speculation
<examine> — adversarial self-critique
<response> — clean synthesisThe multi-model collision array collapsed into a single architecture. Role tokens, no extra parameters.
For the full method:
reaperdoesntknow/DualMind_Methodolgy
doi:10.57967/hf/8184.