Current Architectural Limitations & Core Trade-offs

Before diving into the components, it's crucial to understand the high-level trade-offs between autoregressive (AR) and non-autoregressive (NAR) models like diffusion. The choice between them often comes down to a fundamental decision between optimizing for compute efficiency or data efficiency.

Autoregressive (AR) Models: Optimized for Compute

AR models are highly optimized for computational efficiency, but this comes with limitations.

• Strengths: The sequential, left-to-right process with teacher forcing and causal masking is exceptionally efficient on modern hardware, achieving a high signal-to-FLOPs ratio during training.
• Limitations:
  • Error Propagation & Exposure Bias: An early mistake can't be corrected and often leads to a cascade of errors, degrading the quality of the entire sequence (Tang et al., 2023).
  • Restrictive Inductive Bias: The strict causal (left-to-right) structure prevents the model from learning from the full bidirectional context of the data.
  • Slow Inference: Generating a sequence of length N requires N sequential forward passes, making inference slow.

Diffusion Models (NAR): Optimized for Data

Diffusion models are "super data learners" that trade higher computational costs for a deeper understanding of the training data.

• Strengths:
  • Superior Data Efficiency: By repeatedly training on the same data with different random masks, diffusion models can extract significantly more information from a fixed-size dataset.
  • Bidirectional Modeling: The masking objective allows the model to learn from the full bidirectional context of a sequence, removing the restrictive causal bias of AR models.
• Limitations:
  • High Computational Cost: The diffusion objective is computationally "super-dense," requiring more FLOPs per token during both training and inference.

The Two Core Components

Text diffusion models fundamentally consist of two complementary processes that work together to enable generation:

Core Component 1: The Forward Process (Data → Noise)

The forward process systematically corrupts clean text into a simple, tractable distribution. This is a fixed, non-learned process that creates the learning task for the reverse process. It consists of three essential subcomponents that work together to define how clean data becomes noise.

Core Component 2: The Reverse Process (Noise → Data)

The reverse process learns to invert the forward corruption, step-by-step, to generate new data. This is the learned, generative core of the model. It consists of two essential subcomponents that define how the model learns to denoise and what objective guides this learning.

Core Component 1: The Forward Process

Subcomponent 1A: Input Representation

This subcomponent determines how raw text is converted into a format suitable for the diffusion process.

Discrete Diffusion operates directly on tokenized text, using token IDs from a standard vocabulary (e.g., BPE, WordPiece). Continuous Diffusion converts discrete tokens into continuous vector representations, typically through embeddings or contextualized encodings from pre-trained models.

Subcomponent 1B: The Corruption Process

Discrete Diffusion

The corruption process operates as a fixed Markov chain, starting with a clean text sequence (x₀) and progressively replacing discrete tokens with a special [MASK] token until the sequence is fully degraded. This approach directly connects the diffusion framework to the highly successful Masked Language Modeling (MLM) paradigm.

Continuous Diffusion

This approach first maps tokens into a continuous vector space using contextual encodings from pre-trained models (e.g., BERT). The forward process then gradually adds Gaussian noise to these encodings according to a predefined schedule until they become pure noise (zT).

Shabalin et al. (2025) demonstrate that using contextual encodings is superior to using context-free embeddings, providing the diffusion model with a more suitable latent space for training.

Subcomponent 1C: The Corruption Schedule

Discrete Diffusion

Continuous Diffusion

The schedule defines how much Gaussian noise is added at each step t of the forward process. Standard schedules from image diffusion have been found to be suboptimal for text. Shabalin et al. (2025) propose a tan-d noise scheduler designed to introduce a significantly higher and more consistent level of noise across all timesteps.

Core Component 2: The Reverse Process

Subcomponent 2A: The Denoising Network

Discrete Diffusion

A neural network (typically a Transformer) learns to predict the original tokens at masked positions. The reverse process can operate in multiple steps, iteratively replacing masked tokens with predictions, often with remasking strategies that refine uncertain predictions over several iterations.

Sahoo et al. (2024) introduced confidence-based remasking, where the model iteratively unmasks the most confident predictions while remasking uncertain ones, significantly improving generation quality.

Continuous Diffusion

A neural network learns to denoise the corrupted latent vectors, typically involving predicting either the noise to be removed or the clean latent vectors directly. Shabalin et al. (2025) demonstrate that using a sophisticated encoder-denoiser-decoder architecture significantly improves performance.

Subcomponent 2B: The Objective Function

Discrete Diffusion

Continuous Diffusion

The objective is typically a regression-style loss. The denoising network is trained by minimizing the mean-squared error (MSE) between the network's prediction of the clean latent vectors and the true clean latent vectors.

Advanced Sampling: Multiple Remasking Strategies

LLaDA introduces multiple remasking strategies for different use cases, moving beyond simple random remasking:

Random Remasking (Algorithm 4)

• Pure random selection of tokens to remask at each step
• Simplest approach, good baseline performance
• Use case: Base models, general text generation

Low-Confidence Remasking (Algorithm 5)

• Remask tokens with lowest prediction confidence
• Significantly improves generation quality
• Use case: When quality is more important than speed

Semi-Autoregressive Remasking

• Divide sequence into blocks, generate left-to-right between blocks
• Apply diffusion within each block
• Use case: Instruction-following models, structured generation

Implementation Insight: LLaDA shows that remasking strategy should be task-dependent, with base models preferring confidence-based approaches and instruct models benefiting from semi-autoregressive strategies.

Implementation Considerations for Developers

Choosing Between Discrete and Continuous

Discrete Diffusion

Recommended for developers who:

• Want direct compatibility with existing NLP tokenization pipelines
• Need interpretable intermediate states during generation
• Are working with limited computational resources (generally more efficient)
• Want to leverage existing masked language modeling knowledge

Continuous Diffusion

Recommended for developers who:

• Need fine-grained control over the generation process
• Are working with rich, contextual representations
• Can afford higher computational costs for potentially better quality
• Want to experiment with novel noise injection strategies

Scaling Lessons from LLaDA

Architecture Scaling

• Standard Transformer components (RMSNorm, SwiGLU, RoPE) work well for discrete diffusion
• No special architectural modifications needed beyond bidirectional attention
• Scales to 8B parameters using similar compute budgets as autoregressive models

Training Scaling

• Batch sizes: LLaDA used 1280 (much larger than typical small-scale experiments)
• Learning rates: 4×10⁻⁴ peak (higher than many AR models due to bidirectional objective)
• Optimization: Standard AdamW with cosine decay works well

Advanced Technique: Accelerating Inference with Speculative Sampling

A major drawback of diffusion models is their slow inference speed due to the iterative nature of the reverse process. Speculative sampling, adapted for diffusion models by De Bortoli et al. (2025), offers a promising solution.

The Core Idea: Instead of running the expensive, high-quality "target" model for every single generation step, a faster, lower-quality "draft" model proposes a sequence of future steps. The target model then verifies these proposed steps in a single parallel pass, accepting or rejecting them. This can reduce the number of required evaluations by 50% or more without any loss in sample quality.

Drafting Strategies for Developers

• Independent Draft Model: Use a separate, smaller, and faster diffusion model as the draft model.
• Frozen Target Draft Model: Use the output of the target model from the first step as a "frozen" prediction for all subsequent steps in a window. This requires no extra training and can be implemented out-of-the-box.