Next-Concept Modeling: Rethinking the Token-by-Token Bottleneck

Writer

What if faster text generation required changing the unit of prediction, rather than simply making the model smaller or the hardware faster?
That is the question behind Next-Concept Modeling (NCM). Instead of constructing an answer one token at a time, NCM generates a hierarchy of learned representations, moving from a coarse sketch toward a detailed output. It predicts all positions within each level in parallel, while progressing through the levels sequentially. The result is a much shorter sequential generation path. The research paper describes this as a two-stage language-modeling framework.
The early results are substantial: the authors report up to 31× faster generation than a strong diffusion baseline on summarization, alongside promising performance on Sudoku. But those findings do not establish that NCM is 31× faster than a frontier chatbot, or that it has solved general reasoning. The interesting story is more specific: representation and generation order can change both efficiency and the kinds of problems a model handles well.
Next-Concept Modeling shifts the unit of generation from sequential atomic tokens to a multiscale hierarchy of learned discrete codes. By predicting all positions within each resolution level in parallel, NCM significantly compresses the sequential execution path.
The bottleneck is sequential dependency, not merely model size
An autoregressive language model predicts a distribution over the next token, selects a token, and repeats. Each prediction depends on the prompt and the output generated so far. A token is a unit of text representation, not necessarily a word: words can span multiple tokens, and punctuation or other text fragments can also be tokens. The relevant length for inference is therefore the token count, not the word count.
For a conventional one-token-at-a-time generation loop, an output of length L entails roughly L sequential prediction steps. That dependency chain is the architectural issue NCM targets. The paper’s comparison of generation paradigms contrasts this with diffusion sampling and next-concept generation.
There is an important correction to the common explanation that a model must “process the entire conversation again” for every token. Modern autoregressive transformers normally use a key-value (KV) cache, which stores attention keys and values for previously processed tokens. Subsequent steps reuse those representations instead of recomputing them. The dependency on earlier context remains, but the computation is not a complete restart. Hugging Face’s caching documentation explains this distinction.
Caching does not eliminate the cost of repeatedly applying model weights, accessing cached state, and moving data through the accelerator. The balance between arithmetic work and memory traffic depends on model architecture, batch size, sequence length, numerical precision, and hardware. Efficiently Scaling Transformer Inference demonstrates why inference performance is a systems trade-off, rather than a simple function of parameter count.
Calling a language model “autocomplete” describes its prediction interface, but does not settle what it can learn or whether it can generalize. Likewise, the NCM results do not establish a comparison with human intelligence. Claims about a child’s reasoning, brain memory expressed in bytes, or millions-fold resource differences require separate evidence and a defined comparison. They are not needed to understand the architectural contribution here.
Three generation strategies, three different dependency structures
The useful distinction is not “NCM versus all large language models.” NCM is itself a language-modeling framework and still uses a transformer. The distinction is how the output is represented and generated. Sections 1–3 of the paper establish the following comparison.
| Strategy | What is predicted at each step? | What remains sequential? |
|---|---|---|
| Autoregressive next-token prediction | One next token in the conventional decoding loop | Successive output tokens |
| Full-sequence diffusion or flow-based generation | Updates across many output positions | Successive refinement steps |
| Next-Concept Modeling | All discrete codes at one resolution level | Successive coarse-to-fine levels |

Diffusion language models already offer parallelism. Many refine a full-length sequence repeatedly, but reducing the number of refinement steps can hurt quality. NCM changes both the representation and the schedule: early levels contain fewer positions, so early generator calls need not process the full output length. This is a central distinction in the paper’s motivation and related work.
NCM’s closest image-generation inspiration is next-scale prediction over discrete, multiscale representations, not simply the idea that diffusion always produces progressively higher-resolution images. The paper explicitly connects its design to residual quantization and next-scale image generation. Text introduces a difficulty images do not share in the same way: there is no obvious equivalent of shrinking an image to obtain a lower-resolution sentence. NCM must learn that representation. Section 2 explains the connection and the adaptation challenge.
How NCM works: learn the representation, then learn to generate it
NCM separates representation learning from generation. First, a tokenizer learns to encode and reconstruct text through a hierarchy. Then a transformer learns to generate that hierarchy. The tokenizer is frozen during generator training. Sections 3.3–3.4 describe the two stages.

Stage 1: Encode text as coarse-to-fine residual codes
The tokenizer is a multiscale residual vector-quantized variational autoencoder (VQ-VAE). The important mechanism is vector quantization: continuous latent vectors are represented using entries from learned codebooks. A discrete code identifies a selected codebook entry. Residual quantization then represents what earlier levels failed to capture. Section 3.1 provides the mathematical formulation.
In practical terms, the hierarchy works like this:
- Encode the original text into latent vectors.
- Represent a coarse approximation using a short sequence of codes.
- Calculate the remaining representation error.
- Encode additional corrections at progressively finer levels.
- Combine the levels into a cumulative latent reconstruction and decode it into text.
Each level contributes a correction, not a separate complete answer. The finest level has as many code positions as the target has token positions, but its codes are still latent codebook indices, not ordinary output tokens. The final decoder converts the cumulative representation into token distributions. Sections 3.1–3.4 make this distinction explicit.
A useful analogy is a progressive document preview: the broad structure appears first, then detail becomes legible. But the analogy stops short of claiming that NCM literally writes a human-readable outline. Its “concepts” are learned codes. They are not guaranteed to correspond to explicit ideas, sentences, or a human plan.
Stage 2: Generate one level at a time
At inference time, the generator starts with an empty cumulative reconstruction. For each level, it resizes the reconstruction from earlier levels, predicts the next level’s code distributions, samples all positions in parallel, and adds their contribution. After the last level, the tokenizer’s decoder produces the output text. Algorithm 2 specifies this process.
The architecture is therefore parallel within levels and sequential between levels. A hierarchy with K levels requires K generator evaluations for a basic sample, plus final decoding. Those levels are not the transformer’s internal neural-network layers: the shared generator contains its own transformer blocks and is invoked repeatedly across the representation hierarchy.
Prompt-conditioned generation uses a prefix-language-model attention pattern. Target positions can attend to the prompt and their own level’s input; prompt positions cannot attend to target blocks. This supports the paper’s summarization, translation, and Sudoku experiments. Section 3.4 describes the conditioning mechanism.
Training and inference also differ. During training, the target hierarchy is available, allowing all levels to be predicted in one forward pass with appropriate attention restrictions. During generation, later levels depend on earlier generated codes. To reduce error accumulation, the researchers corrupt some conditioning codes during training while retaining clean targets. Moderate corruption improves robustness; excessive corruption weakens the conditioning signal. Appendix F.3 reports that trade-off.
What the speed results actually show
The paper was submitted on 9 October 2026 by researchers affiliated with Stanford University and Together AI. The findings discussed here are from its initial arXiv version, checked against the paper and released repository as of 27 October 2026. They should be read as reported research results, not independently reproduced measurements. Publication record.
The authors report three headline comparisons against strong diffusion baselines at comparable generator scales:
- LM1B: 21× faster generation, with better generative perplexity than DUO.
- OpenWebText: 10× faster generation, with competitive quality against the strongest diffusion baseline.
- XSum summarization: 31× faster generation than AURORA-LM at comparable ROUGE quality.
These results are described in the abstract and efficiency analysis.
The 31× figure is a task-specific comparison with a diffusion baseline. It is not a measured 31× advantage over every autoregressive model, a frontier assistant, or an arbitrary production workload.
The wall-clock benchmark generates 1,024 sequences at batch size 16 on one NVIDIA H100 SXM 80 GB GPU. That is 1,024 samples, not necessarily 1,024 tokens per sample. Separately, the OpenWebText configuration generates 1,024-token targets using 32 hierarchy levels. XSum uses 48-token targets and eight levels. These are different experimental settings and should not be merged into a single “1,000-token summarization” claim. Section 4.6 and Appendix C provide the conditions.
The paper also distinguishes quality measurements from timing measurements. Some conditional baseline quality scores come from prior publications; for baselines without released checkpoints, timing uses the reported architecture and sampling configuration with randomly initialized weights. Those runs measure execution time, not generation quality. Appendix D.2 documents the methodology.
Quality is not one number
For unconditional generation, generative perplexity measures how probable generated text appears to a reference language model. Lower is generally better, but repetitive or less diverse output can also score well. The paper therefore reports unigram entropy alongside perplexity and compares models with diversity near that of the underlying data. Section 4.2 explains this evaluation choice.
Summarization uses ROUGE overlap metrics, while translation uses BLEU. These metrics support benchmark comparisons, not a blanket claim of factual reliability. The paper’s qualitative samples include incorrect entities and altered details, showing why fluent output and competitive overlap scores are insufficient evidence of correctness. Sections 4.3 and 4.7 and Appendix J provide the results and examples.
Fewer passes do not automatically mean less total work
A simple thought experiment captures the appeal. Assume a dense generator has 1 GB of weights, each evaluation reads those weights once from accelerator memory, and there is no batching, cache reuse of weights, or other traffic. A conventional 1,024-token decoding loop would imply about 1,024 GB of weight reads; a 32-level generator would imply about 32 GB. The ratio is 32×.
This is illustrative weight-traffic accounting, not a measured NCM memory-bandwidth result. It excludes attention state, activations, latent reconstruction, final decoding, hardware caching, and the differing work performed by each call. It also describes bytes transferred, not resident memory capacity. Both hypothetical models still have 1 GB of weights.
The complication is that NCM predicts multiple code positions per call and processes codes across several levels. The total number of code positions can exceed the number of final text tokens. However, even a hierarchy containing four times as many codes would not establish exactly four times the computation: attention shapes, prompt processing, codebook operations, and decoder work all matter. The paper explicitly warns that number of function evaluations (NFE) alone does not capture inference efficiency. Section 4.6.
The more defensible interpretation is that NCM changes the workload’s shape. It shortens the sequential path and concentrates more parallel work into each step, while keeping early levels small. Whether that shifts a particular implementation from memory-bound to compute-bound requires profiling. The paper demonstrates wall-clock gains against its baselines; it does not establish a universal compute multiplier, energy saving, or water-consumption reduction.
Sudoku: evidence for global refinement, not general intelligence
Sudoku is a revealing test because every cell must agree with row, column, and box constraints. A locally plausible choice can invalidate the entire board. NCM’s ability to refine a representation of the whole output is a plausible architectural advantage for this kind of task. The authors offer that interpretation in Section 4.4.
Their reported results are percentages of completely solved boards:
| Model or decoding method | Generator size | Generator evaluations | Easy | Medium | Hard |
|---|---|---|---|---|---|
| Autoregressive, greedy | 25M | 128 | 14.6% | 5.1% | 1.0% |
| DUO | 29M | 128 | 96.3% | 84.7% | 58.4% |
| FMLM+ | 29M | 16 | 97.7% | 92.8% | 81.3% |
| NCM | 25M | 12 | 99.5% | 96.5% | 86.1% |
Source: Table 3. Difficulty is defined by 40, 35, and 30 provided clues, respectively; baseline results are taken from prior work.

The hard-board comparison is striking, but describing it as “86 boards versus one out of 100” would turn percentages into an unsupported sample count. Likewise, an enormous relative improvement over a 1% baseline obscures the stronger comparison: NCM achieves 86.1% versus FMLM+‘s 81.3%, using 12 rather than 16 evaluations.
The reasonable conclusion is that generation order and representation matter for globally constrained outputs. The experiment does not demonstrate superior general reasoning, reasoning without training data, or superiority over frontier assistants using tools, search, or verification. It evaluates small, task-specific models under a particular protocol.
The remaining catches
Reconstruction is another source of error
The tokenizer must preserve text accurately before generation quality even enters the picture. The paper reports token-level reconstruction accuracy of 99.99% on OpenWebText, 99.79% on XSum, 98.79% on WMT14, and 100% on the evaluated Sudoku data. Those are reconstruction results, not generated-answer accuracy. Table 1.
Rare errors still matter. A substituted name, numeral, or markup token can change meaning or break an exact-output task. Appendix G documents these failure categories. A near-lossless representation is encouraging, but is not interchangeable with guaranteed lossless encoding.
The schedule is fixed
NCM fixes hierarchy depth and per-level lengths. Even a short continuation traverses the hierarchy. The authors identify input-dependent schedules and end-to-end training as future directions, rather than implemented capabilities. They also report occasional artifacts around coarse-level transitions. Appendix I.
Intermediate decoded outputs are not guaranteed to improve monotonically. A later level can temporarily worsen a decoded sample before subsequent corrections improve it. The progressive-refinement analogy is useful, but not a promise that every intermediate answer is better than the last. Appendix J.2.
Scaling evidence is promising but limited
Most main experiments use generators around 100 million parameters. The LM1B scaling study tests 97M, 317M, and 725M generators sharing a frozen tokenizer, with improved quality and diversity at a fixed generator training-token budget. The largest reported generator is therefore 725M, not precisely 730M, and its parameter count excludes the separate tokenizer. Section 4.5.
That is evidence of useful scaling within the tested range. It does not establish that the same advantages persist at multi-billion-parameter or frontier scale, on much broader datasets, or in long-running interactive workloads. Nor is it necessary to assume undisclosed frontier systems all have trillions of parameters to frame the open question.
Training efficiency needs careful accounting
On LM1B, tokenizer and generator training together expose NCM to about 13.36 billion full-length token positions, compared with 66 billion for the diffusion baselines. The paper reports roughly four to five times fewer training tokens on LM1B and OpenWebText, including tokenizer training. These budgets include repeated exposure and padding; they are not measurements of unique data or total floating-point operations. Section 4.6 and Appendix C.2.
Why the architecture is worth watching
NCM’s efficiency can be spent on more than lower latency. The researchers also generate multiple candidates and select one using minimum Bayes risk (MBR) decoding, here choosing the candidate with the highest average similarity to the others. On XSum, this improves benchmark quality at matched inference time. It is candidate selection, not an independent factual verifier. Section 4.5.
For technically informed readers, the next evidence to look for is straightforward:
- Independent reproduction under the published benchmark conditions.
- Larger-scale comparisons with strong autoregressive and diffusion systems at matched quality.
- Exactness and factuality evaluations beyond overlap metrics.
- End-to-end profiling that includes tokenizer decoding, memory traffic, and varying batch sizes.
- Tests of variable-length and interactive generation rather than only fixed task configurations.
The released repository includes evaluation code, configurations, and links to checkpoints and datasets. That makes the research inspectable, but should not be confused with a production-ready general-purpose assistant.
The real takeaway: prediction units are a design choice
NCM does not establish that next-token models are obsolete. It demonstrates something more useful: token-by-token generation is not the only way to organize language modeling.
By learning a coarse-to-fine representation, NCM reduces sequential generator calls and gives the model a different way to coordinate an entire output. Its benchmark speedups and Sudoku results justify attention. Its small scale, fixed schedules, reconstruction errors, and limited evaluation scope justify restraint.
The important question is not whether one early paper will dethrone today’s assistants. It is whether better representations and generation schedules can move the quality–efficiency frontier. NCM offers concrete evidence that they can, within the settings tested. Whether that leads to much broader capability, or eventually contributes to artificial general intelligence, remains speculation rather than a finding of this research.
Read next


