arxiv:2604.11748

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Published on Apr 15

· Submitted by

Chumeng Liang on Apr 16

University of Illinois at Urbana-Champaign

Upvote

Authors:

Abstract

LangFlow demonstrates that continuous diffusion models can match discrete counterparts in language modeling by leveraging embedding-space flow matching with novel training techniques and noise scheduling.

AI-generated summary

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow

View arXiv page View PDF Project page GitHub 14 Add to collection

Community

chumengl

Paper submitter about 4 hours ago

Continuous diffusion has dominated image and video generation — but in language modeling, it has long been considered inferior to discrete diffusion.

We challenge this belief with 🚀LangFlow: the first continuous diffusion language model that rivals — and even surpasses — discrete diffusion.

📝 Blog: https://caradryanl.github.io/blog/2026/langflow/
💻 GitHub: https://github.com/nealchen2003/LangFlow
💡 HuggingFace: https://huggingface.co/Continuous-Rivals-Discrete
📄 Arxiv: https://arxiv.org/abs/2604.11748

LangFlow shows for the first time that continuous diffusion can rival discrete counterparts on language modeling:
🔍 On LM1B and OpenWebText, both our perplexity (PPL) and generative perplexity (Gen. PPL) match or surpass the best discrete diffusion models.
🔍 On zero-shot transfer, LangFlow outperforms the best discrete diffusion on 3 out of 7 and autoregressive baselines on 4 out of 7 benchmarks.

LangFlow connects embedding-space DLMs to Flow Matching. It predicts clean token probabilities from noisy token embeddings, and then derives the embedding-space flow in closed form. Endorsed by Bregman divergence, we train the model via the cross-entropy loss.

However, training such continuous diffusion on language has long been a struggle. We thereby reveal several crucial ingredients:

📌 The noise schedule should make the information gain per unit time uniform. Under this principle, the optimal noise scheduler for language is a Gumbel distribution, greatly different from that for images.

📌 Self-conditioning significantly improves likelihood and sample quality, with effects substantially different from discrete diffusion. Disabling it during comparison with discrete DLMs is unfair. The protocol of training and evaluating continuous DLMs should be rectified!

📌 Using ODE sampling, LangFlow can naturally derive a novel ODE-based likelihood bound from Flow Matching, enabling principled evaluation by perplexity, the core metric of language modeling.

The potential of continuous DLMs extends far beyond just performance. They open the door for all continuous diffusion techniques to be introduced into language modeling:
-- One-step generation, such as Consistency Models
-- Guided generation, such as CFG
-- Unified multimodal generation, such as protein structure-sequence co-design

🌱 LangFlow suggests: continuous diffusion is a viable and promising paradigm for language modeling!

I'm grateful to be part of such an amazing team pushing this forward. A huge thank you to our great advisors Ge Liu and Jiaxuan You, and to our amazing collaborators Chumeng Liang, Yuxin (Neal) Chen, Ruihan Guo, Chaoran Cheng.