Keep Drafting Parallel — Inco AI

Inference is the bottleneck of the agent era. Agents read, plan, and call
tools, often for hours or days. They consume tokens at a rate chat never
approached. Every one of those tokens takes a full forward pass over the
model. At Inco AI, we are building the inference stack scaled to the token
economics of tomorrow. This post is a sneak peek.

Our team released DFlash in January; it
now runs in SGLang, vLLM, TensorRT-LLM, and llama.cpp. NVIDIA measured
up to 15× throughput
with it on Blackwell GPUs; Google reported
3× more tokens per second
on TPUs; CoreWeave’s production Kimi K2.7 Code endpoint, the fastest for
that model on Artificial Analysis,
runs DFlash by default. The ecosystem now builds on it:
NVIDIA,
Red Hat, and
Modal have all published
DFlash drafters; Meta
(Muse Glimmer),
Poolside (Laguna),
Xiaomi
(MiMo-V2.5-Pro),
and NVIDIA
(Nemotron 3.5 Lightning)
ship official drafters with their own models. On Hugging Face, DFlash models have
been downloaded more than 3.5 million times (as of August 2026).

Speculative decoding is a core piece of the modern inference
stack.¹ A small draft model guesses a block of tokens,
and the target model verifies the whole block in one forward pass. Good guesses
turn one pass into several tokens; bad ones just get thrown away. For years, though, the draft itself stayed autoregressive:
one token at a time. DFlash made it one-pass too: the entire block, every
position, predicted in parallel.

DFlash 2 pushes parallel drafting one step further: over 20% more output from
every verification pass, for around 1% added cycle latency, with the output
provably unchanged. Across benchmarks the gain runs 16–25%. With the
Qwen3.8-27B drafter released today, SGLang serves at 2.7–3.4× the
throughput of autoregressive decoding at batch size 1. Predicting every
position independently leaves headroom in two places: choosing the right
tokens and holding accuracy to the end of the block. DFlash 2 recovers
both without giving up the one-pass design.

DFlash 2 already runs in the mainstream inference engines:

pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git#subdirectory=python"
 
python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 \
  --speculative-num-draft-tokens 8

pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/52816/head"
 
vllm serve Qwen/Qwen3.8-27B \
  --speculative-config '🔥'

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
git fetch origin pull/27342/head:pr-27342
git switch pr-27342
 
# NVIDIA CUDA
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j
 
# Apple Silicon
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON
cmake --build build -j
 
./build/bin/llama-server \
  -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \
  -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \
  --spec-type draft-dflash \
  --spec-draft-n-max 7

DFlash predicts every position independently, in parallel. Each pick is
plausible on its own. Yet nothing makes them fit together, and an
incoherent block is cut short at verification.
Recent methods such as Domino and
DSpark buy coherence with sequential
heads that rewrite each position’s full-vocabulary distribution.
But is that costly autoregressive correction really necessary?

No. The evidence is already in DFlash’s own candidate lists. Take the
first position: DFlash’s top pick is right 85.4% of the time, but the
right token is in its top 16 candidates 99.5% of the time. Even when the top pick
is wrong, the right token is usually on the list.

An oracle that always picks the right candidate from the top 16 would
lift the acceptance length from 4.27 to 6.79. That gap is pure selection
headroom. We just need to select the right path through the candidates.

A Lightweight Path Selector

Coherence is mostly local: a candidate’s fit depends mainly on the token
just before it, so scoring neighboring pairs should be enough. DFlash 2
keeps the top 16 candidates at each position and scores every adjacent
pair: for predecessor $a$ and current candidate $b$,

$$S_t(a,b)=U_t(b)+\langle A(a)\odot H(h_t),B(b)\rangle.$$

The score has two parts. The first, $U_t(b)$

Scoring stays fully parallel. Every adjacent pair at every position is
scored in one shot, with no extra backbone or LM-head pass. The only
sequential work is the final walk over precomputed scores: starting from
the last verified token, greedy follows the best successor at each step,
sampling draws from the same scores, and rejection sampling restores the
exact target distribution.

The selector improves DFlash by 0.34 tokens at $T=0$

We also noticed
both recall rows above decline toward the end of the block.
Even the oracle decays: with perfect selection, accuracy still falls from
99.5% at the first position to 87.8% by the last. No selector can fix
that, because the candidates themselves are running out. We call this
suffix decay, and it is a backbone problem.

One suspect is capacity: a five-layer backbone may be too small to
preserve dependencies across the block. If that is right, depth should
help most at later positions. And it does!
3-, 5-, and 15-layer DFlash models are almost identical at the first
position, and fan apart down the block. But depth is indiscriminate: ten
extra attention blocks add capacity everywhere, even at the
early positions that had little left to gain, and erase much of the
efficiency that makes DFlash attractive.

We want a targeted fix, and DFlash’s attention shows where. It has two
jobs: read the context before the block, and model the dependencies
inside. But it spends less and less on the second: the block’s
share of attention falls from 30% in Layer 1 to 8% in Layer 5, and
what remains concentrates in a shrinking handful of heads. So we split the
jobs: a dedicated module takes the within-block work, and attention keeps
reading the context.

A Lightweight Local Convolution

The within-block work is short-range to begin with: a block spans only
4 to 16 tokens, and the tightest dependencies sit between neighbors. The
natural operator is a short convolution: two taps, one on the current
position and one reaching one position back, with weights that adapt to
the content. Following
Canon Layers,
Dynamic Short Convolutions, and
Convolution for Large Language Models,
we insert this two-tap dynamic depthwise convolution before and after each
attention and feed-forward sublayer:

$$\operatorname💬_💬(x)_t =k_💬\odot x_t+k_{t,1}\odot x_{t-1}.$$

Each coefficient combines a learned base kernel with a small correction
computed from the current hidden state; every 16 channels share one
correction. The first position reads the last verified token’s
representation, and every later position reads its predecessor’s.
Information crosses the block while all positions still compute in
parallel.

The convolution is block-local and stateless, so it drops into DFlash without
changing attention, the LM head, or verification.

With only 16.5M added parameters (3%), five-layer DFlash with
convolution comes close to 15-layer DFlash, substantially
reducing suffix decay. The convolutions add 0.7% to draft–verify cycle
latency; ten more Transformer layers add 15.2%. Average within-block
attention across Layers 4 and 5 also falls from 9.4% to 0.5%,
consistent with the convolution absorbing the local work while attention
goes back to reading the context. A kernel reaching one position back
recovers most of what ten extra layers buy: suffix decay is mostly a
local problem.

So far, the selector and the convolution have been measured separately;
the full comparison below puts them together. We trained the DFlash
and DSpark drafters ourselves under matched setups, while MTP ships with
the model.

DFlash 2 leads on every benchmark. Averaged across them, it gains
1.05 tokens over DFlash (21%) and 0.48 over DSpark. The upgrade
stays cheap: the selector and the convolution together add only 1.3%
to the five-layer DFlash draft–verify cycle latency.

On MATH-500, the gain is visible position by position:
DFlash 2 holds steady near 86% to the last position, and every baseline
ends the block 6 to 9 points below it.

We are releasing two DFlash 2 drafters today:
one for Qwen3.8-27B
and
one for Meta’s Muse Glimmer.
For Qwen3.8-27B, we compare against the model’s native MTP path and a
community DSpark drafter.

For Meta’s Muse Glimmer, we compare against the official DFlash drafter
shipped with the model and a
community DSpark drafter.

The margins are wide: on both models, DFlash 2 averages more than
a full token ahead of DSpark. It also beats each model’s official
drafter, MTP on Qwen3.8-27B and DFlash on Muse Glimmer. That translates
into 2.7–3.4× the throughput of autoregressive decoding on
Qwen3.8-27B, and 3.1–4.6× on Muse Glimmer. The
model cards break the speedups down
by task and concurrency.

An agent writes in an afternoon what a chatbot writes in a month, and
decoding sits under every one of those tokens. DFlash 2 decodes at
close to 3× the speed of autoregressive decoding, about a third of the
compute per token, with the same output.

In seven months, DFlash went from our paper to an industry standard,
with more than 3.5 million downloads. Inside the same design, DFlash 2
decodes one more full token per pass, for free. That is only one component of the serving stack.
Inference is nowhere near its floor.

At Inco AI, we are building an end-to-end serving stack to keep pushing that
floor lower. DFlash 2 is the first piece. Two drafters are out today
on Hugging Face.

If you serve agents at scale and want to evaluate DFlash 2 in your stack,
or want a drafter for a model you run, including your own fine-tunes, write
to us: contact@inco.ai.

We are also hiring. If you want to help build this stack, reach out to
us.

Connect the candidates. Keep drafting parallel.

Please cite this post as:

@misc{inco2026dflash2,
  title  = {{DFlash 2: Keep Drafting Parallel}},
  author = {{Inco AI}},
  year   = {2026},
  month  = {August},
  url    = {https://inco.ai/blog/dflash2/}
}