Speculative decoding is an AI speedup technique where a small, fast "drafter" model guesses upcoming words. It speeds up text generation by letting the main AI review and accept multiple words at once.

The trusted old method is MTP. It works with stock llama.cpp and makes everything way faster with ~zero quality loss. Everyone loves MTP.

However, DFlash is a new fast speculative decoding method for LLMs and many people are raving about it, so I decided to investigate.

This is preliminary research with no optimizations. I'm trying to find out why the results are the way they are and how to improve them.

If you don't care about the test and methodology, skip to Test results or Conclusion

Preliminary findings: Confusing

I used my two favorite local models: the 27b and 35b versions of the Qwen 3.6 family. The results were... mixed at best.

On Qwen 27b with 3 MTP heads, I usually get ~18 tokens/s. With 6 DFlash heads, I got ~20.5 tok/s. Not bad. Free performance, but I decided to explore further.

Then things got awkward. I tried the Qwen 35b MoE, which got slower. Much slower. From ~59 tok/s down to 41.5 tok/s. Huh?

The Vulkan vs ROCm thing

Or, everything is weird in AMD land.

It suddenly hit me that I was using the community-supported Vulkan backend, which is usually on par or better than the "official" ROCm one on LLM workloads.

Since everyone seems to be optimizing for the new kid ROCm 7.14 instead of Vulkan, I thought switching might make the difference.

And it did. But it was still weird.

Setup

For MTP, I use 3 drafters for every model. This is the best mix of speedup and resource usage for my needs

For DFlash, I use 6 drafters. Why 6? I have no idea. I went by what randos on the internet wrote and vibes. I'm still experimenting.

Methodology

Ask two questions to each model, one after the other:

  1. Hey, who are you and what are your capabilities?
  2. Sweet. How good are you at coding?

This might not seem representative, but it gives a good idea of how models will behave when context grows larger. Almost every mid-size open model I've tried tends to reason a lot even for simple questions, so the total output is usually 2000+ tokens including thinking.

My armchair theory is that the small models are tuned to reason longer to make up for the lower parameter count. The models stumble over themselves for a while until they reach the correct answer.

Test results

As I said, the results are all over the place. Except for two combos, everything is usable.

Q1 and Q2 are the speeds I got for the first and second question respectively.

Qwen 3.6 35B - Unsloth Q8 K_XL

ROCm DFlash = ROCm MTP = Vulkan MTP > Vulkan DFlash

Drafter ROCm 7.14 Q1 ROCm 7.14 Q2 ROCm tokens generated Vulkan Q1 Vulkan Q2 Vulkan tokens generated
DFlash 75.1 tok/s 57 tok/s 614 + 1491 65 tok/s 56.2 tok/s 570 + 1385
MTP 70.1 tok/s 61 tok/s 521 + 1063 71.5 tok/s 59.6 tok/s 621 + 985

Qwen 3.6 27b - Unsloth Q8_0

Vulkan DFlash > ROCm DFlash > ROCm MTP = Vulkan MTP

Drafter ROCm 7.14 Q1 ROCm 7.14 Q2 ROCm tokens generated Vulkan Q1 Vulkan Q2 Vulkan tokens generated
DFlash 24.2 tok/s 19.8 tok/s 688 + 1149 25.1 tok/s 20.23 tok/s 883 + 1154
MTP 18.2 tok/s 18.1 tok/s 1047 + 1539 19.8 tok/s 17.7 tok/s 1241 + 1054

Gemma 4 31b - Unsloth Q8_0

Vulkan MTP > ROCm DFlash = ROCm MTP > Vulkan DFlash

Drafter ROCm 7.14 Q1 ROCm 7.14 Q2 ROCm tokens generated Vulkan Q1 Vulkan Q2 Vulkan tokens generated
DFlash 20.3 tok/s 15.6 tok/s 839 + 1006 19.7 tok/s 15.7 tok/s 960 + 1181
MTP 19.7 tok/s 16.3 tok/s 854 + 1055 20.7 tok/s 16.5 tok/s 955 + 1133

Gemma 4 31b QAT - LM Studio Community Q4_0

This is the only unusable combo. Do not use DFlash on Gemma 4 31b QAT. It's objectively worse in all tests. In fact it's even slower than the Q8 model. Just use MTP.

ROCm MTP = Vulkan MTP >> ROCm DFlash > Vulkan DFlash

Drafter ROCm 7.14 Q1 ROCm 7.14 Q2 ROCm tokens generated Vulkan Q1 Vulkan Q2 Vulkan tokens generated
DFlash 21.1 tok/s 15.2 tok/s 928 + 1055 ~15.5 tok/s ~12 tok/s 1059 + 1064
MTP 28.3 tok/s 24.7 tok/s 961 + 1227 ~28.5 tok/s ~25.5 tok/s 1026 + 1124

Conclusion

Different models behave very differently depending on the backend and spec decoding method. No "correct" answer, but the consensus is:

Good choices

  • Qwen models: ROCm + DFlash
  • Gemma models: Vulkan + MTP

Terrible choices

  • Qwen 35b + Vulkan + DFlash
  • Gemma 31b QAT + anything DFlash (this one is especially terrible. Avoid at all costs)

Nonoptimal but usable choices

  • Everything else

More to come soon.