The really interesting thing to me was that they didn’t need to train this model from scratch they just used their existing MOE checkpoint:
“To convert a decoder-only model (Gemma 4 26B A4B) into a denoiser, we can make use of something it is not directly using when generating tokens, namely the logits of all tokens!”
What makes me hopeful about this release is that possibly this same conversion can be applied to other open models and we might see a bunch of diffusion versions of existing local models. It’s exciting stuff!
That also surprised me, the diffusion gemma is actually a bunch of architectural changes to make discrete diffusion practical and an sft + distill of the Gemma 4. One cool trick they do is to take advantage of the entropy of the estimated probability distribution to adaptively adjust the required number of diffusion steps for inference. Image diffusion only yields the score function, a gradient instead of the probabilities, so the same trick does not apply.
I like the model a lot and it's fairly good at reasoning. You can also really bend it to your needs. It's designed for machines with more compute than memory bandwidth but IMO does really well on metal.
I've got it up to ~15tok/s on M3-class machines, but I wager there's a bunch of perf on M5 that I just don't have hardware access to unlock.
I tried to implement MTP using the other Gemma MTP heads but I failed to move that perf needle. There's some interesting research to be done about pre-seeding the diffusion canvas from draft models. DiffusionGemma with the right drafter can hit 20-30 tok/s on my machine, but I've been unable to combine the two together to make it faster than what it's been running at so far.
I've also re-implemented DiffusionGemma from scratch! But with Rust+CUDA rather than macOS, I'm sure our club is a fairly small one :)
About the drafting/MTP, how would that work? As I understand, MTP/drafting is mostly useful for autoregressive models, not for a diffusion model, because each canvas (in the case of DiffusionGemma at least) is fast enough, what you care about is the "convergence" time before it figures out that the accuracy is good enough to move on to another canvas. This is my understanding at least. Would this drafting for a diffusion model like this mean you'd generate multiple canvas at the same time? That'd also seem strange, as then you'll cut performance for generating a complete canvas... I'm genuinely curious about how it could be applied to DiffusionGemma though, so please do expand :)
You can save steps when diffusing from a pre-drafted canvas instead of seeding with random noise! If you have a drafter that is good enough, steps drop by ~50% or more, and tok/s can definitely increase.
I'd be curious to see your CUDA approach - happy to combine efforts as well.
That's mostly correct, but I believe the INT8 support on M5 changes the equation pretty drastically. I also believe that diffusion models generally output _better_ results if they have shaping for code (they often will miss delimiter matching inside of a canvas).
If these models get good at coding it's going to force a rethink of how languages, compilers and test suite runners work. "AI changes everything" is a cliché by this point
but I think it's actually true.
If your model can reason and write code at 1500 toks/sec, then you should end up totally bottlenecked on CPU time the entire time a prompt is active. If you aren't, then you're losing wall time versus competitors. But our whole development stack is based around the idea that programmers spend most of their time thinking, talking and coding, not waiting for the CPU (melting CI clusters being a painful exception to that).
What I'm imagining here is some sort of hybrid mode in which compiling code and running it through an interpreter can be overlapped, so an LLM can propose a change and immediately begin running unit tests while type errors that might affect some other module are found in parallel. And the tests would always run sharded, potentially on a remote cluster, even in local dev.
Obviously this approach is to some extent what made the JVM popular. Java compiles very fast because javac does little more than type checking, and the type system is simple. Then the JVM does the heavy lifting of compilation in parallel with it running. So although Java has a reputation for poor startup times, turnaround times for the JVM can be really excellent compared to something like C++, Swift or Rust. And a lot of startup time pain is just poor frameworks like old Springs that want to reflectively scan the app's files and do other inefficient stuff. More modern frameworks push more to incremental build tasks and can get startup down to very little, <0.5secs for a web server with DB connections to start for instance.
But it feels like this approach should be pushed much further. The model should spend all its time waiting on unit tests to run.
LLMs are excellent at reading stack traces, better than humans. The underlying algorithm lets them link every token with every other in parallel (modulo approximations, linear attention, etc). And they're trained on needle-in-a-haystack tasks. You really want to present LLMs with lots of data up front in large chunks to get the best out of their algorithms.
I'm very interested in Diffusion text models. The concept of taking noise and adding words starting randomly all over the response, and filling in the noise from there on breaks my brain.
I'm sure I have a fundamental misunderstanding of the technology, though.
Diffusion text models for me is the more interesting type of LLMs for local usage, as it really makes good use of single GPUs for single responses, rather than auto-regressive ones, and is a lot faster! Probably the fastest model I've been able to run so far, ending up doing ~670 tok/s (depending on the type of text) on a Pro 6000
Denoising is probably the weakest part of this model. There is recent research from Kaiming He showing that predicting the noise is actually not the best strategy for image generation since noise space is so large.
Simply predicting the surface of the data you are trying to generate is far more representationally efficient, and perhaps in the next few months we'll see a version of that for LLMs
Diffusion language models work with a discrete output space, unlike image models that repeatedly refine a continuous output, so they don't do the noise-prediction thing anyway.
Normally people have feelings about things before they are able to put them into words, I would imagine if you were asked a question like "what city would you most like to visit" then unless you've already thought about it a lot, then you would have to do substantial non-verbal thinking before you can come up with an answer, and once you have the answer you may respond "my favorite city is X" and you decided what X would be before you started the sentence.
Here's the best way I can explain how I "think" because it is mostly passive, but sometimes, I am actively thinking.
If I'm asked a hard question, like what city I'd most like to visit, I kind of have to look up cities and cross reference them with ones I've noted that I want to visit, or have visited and liked enough to go back, then actually read the list, and figure out which one, but each time I read a city, I might have a, this would be great in summer or winter or they have a cool festival in June. Which leads to additional question I have to ask. Which I think about roughly the same speed I speak, maybe a word or two faster.
My wife is the complete opposite. She can put together a sentence in an instant in "parallel" so when she says it, she though it already, but not as a sentence.
Human brains all work so crazily weird, and we are only just now started to realize all the nuances.
From what I've noticed at least for myself, there are like at least two distinct parts that don't always cooperate. There's a draft part that's random thoughts that just appear and can be converted immediately to speech, but when writing it's actually another part that takes dictation from that one, albeit silently.
If I'm just typing blind I'll somehow end up writing random homophones down with completely correct spelling, like hear instead of here, it's bizarre. So that part is sort of structured autogenerated noise that is then consciously either appended or inserted into random spots in text. Maybe it's more of an LLM first pass then diffusion refinement.
Appealing results... do we think there is scope to close the accuracy gap against AR models? or even leverage the "Bidirectional Reasoning and Self-Correction" into an overall advantage?
DFlash 2 is a diffusion-based speculative decoding head for autoregressive models; there's nothing to accelerate here, because this is already wholly diffusion.
(to finish your thought) Which is important for consumer hardware to be better suited to running these models. Cloud providers are already able to batch as many requests as they want together to improve resource utilization, so they will not see a big benefit from diffusion models.
RDNA4 is pretty similar to CDNA4, yet over a year after the release of "pro AI" cards like the r9700, they had basic kernels lacking in vllm (like w4a16 int4 kernels) while they were implemented in the datacenter CDNA4 cards.
AMD hardware runs well on llama.cpp because basically anything runs on llama.cpp, especially with vulkan. It's not high praise of AMD's software team to say llama.cpp runs well on their hardware
Just wanted to share this, I found it was a really nice resource to understand how diffusion Gemma worked: https://newsletter.maartengrootendorst.com/p/a-visual-guide-...
The really interesting thing to me was that they didn’t need to train this model from scratch they just used their existing MOE checkpoint:
“To convert a decoder-only model (Gemma 4 26B A4B) into a denoiser, we can make use of something it is not directly using when generating tokens, namely the logits of all tokens!”
What makes me hopeful about this release is that possibly this same conversion can be applied to other open models and we might see a bunch of diffusion versions of existing local models. It’s exciting stuff!
That also surprised me, the diffusion gemma is actually a bunch of architectural changes to make discrete diffusion practical and an sft + distill of the Gemma 4. One cool trick they do is to take advantage of the entropy of the estimated probability distribution to adaptively adjust the required number of diffusion steps for inference. Image diffusion only yields the score function, a gradient instead of the probabilities, so the same trick does not apply.
Oh wow that’s really cool!
I re-implemented this one for macOS over the last couple of months: https://github.com/mmastrac/diffgemma
I like the model a lot and it's fairly good at reasoning. You can also really bend it to your needs. It's designed for machines with more compute than memory bandwidth but IMO does really well on metal.
I've got it up to ~15tok/s on M3-class machines, but I wager there's a bunch of perf on M5 that I just don't have hardware access to unlock.
I tried to implement MTP using the other Gemma MTP heads but I failed to move that perf needle. There's some interesting research to be done about pre-seeding the diffusion canvas from draft models. DiffusionGemma with the right drafter can hit 20-30 tok/s on my machine, but I've been unable to combine the two together to make it faster than what it's been running at so far.
I've also re-implemented DiffusionGemma from scratch! But with Rust+CUDA rather than macOS, I'm sure our club is a fairly small one :)
About the drafting/MTP, how would that work? As I understand, MTP/drafting is mostly useful for autoregressive models, not for a diffusion model, because each canvas (in the case of DiffusionGemma at least) is fast enough, what you care about is the "convergence" time before it figures out that the accuracy is good enough to move on to another canvas. This is my understanding at least. Would this drafting for a diffusion model like this mean you'd generate multiple canvas at the same time? That'd also seem strange, as then you'll cut performance for generating a complete canvas... I'm genuinely curious about how it could be applied to DiffusionGemma though, so please do expand :)
You can save steps when diffusing from a pre-drafted canvas instead of seeding with random noise! If you have a drafter that is good enough, steps drop by ~50% or more, and tok/s can definitely increase.
I'd be curious to see your CUDA approach - happy to combine efforts as well.
I have M5, but diffusion is a bad fit for metal: https://eamag.me/2026/why-parallel-diffusion-llms-are-slow-o...
That's mostly correct, but I believe the INT8 support on M5 changes the equation pretty drastically. I also believe that diffusion models generally output _better_ results if they have shaping for code (they often will miss delimiter matching inside of a canvas).
If these models get good at coding it's going to force a rethink of how languages, compilers and test suite runners work. "AI changes everything" is a cliché by this point but I think it's actually true.
If your model can reason and write code at 1500 toks/sec, then you should end up totally bottlenecked on CPU time the entire time a prompt is active. If you aren't, then you're losing wall time versus competitors. But our whole development stack is based around the idea that programmers spend most of their time thinking, talking and coding, not waiting for the CPU (melting CI clusters being a painful exception to that).
What I'm imagining here is some sort of hybrid mode in which compiling code and running it through an interpreter can be overlapped, so an LLM can propose a change and immediately begin running unit tests while type errors that might affect some other module are found in parallel. And the tests would always run sharded, potentially on a remote cluster, even in local dev.
Obviously this approach is to some extent what made the JVM popular. Java compiles very fast because javac does little more than type checking, and the type system is simple. Then the JVM does the heavy lifting of compilation in parallel with it running. So although Java has a reputation for poor startup times, turnaround times for the JVM can be really excellent compared to something like C++, Swift or Rust. And a lot of startup time pain is just poor frameworks like old Springs that want to reflectively scan the app's files and do other inefficient stuff. More modern frameworks push more to incremental build tasks and can get startup down to very little, <0.5secs for a web server with DB connections to start for instance.
But it feels like this approach should be pushed much further. The model should spend all its time waiting on unit tests to run.
no way llms can reason through (spring) java's stacktrace hell, and rust compilation is just too slow, i think golang is gonna be gold.
LLMs are excellent at reading stack traces, better than humans. The underlying algorithm lets them link every token with every other in parallel (modulo approximations, linear attention, etc). And they're trained on needle-in-a-haystack tasks. You really want to present LLMs with lots of data up front in large chunks to get the best out of their algorithms.
I'm very interested in Diffusion text models. The concept of taking noise and adding words starting randomly all over the response, and filling in the noise from there on breaks my brain.
I'm sure I have a fundamental misunderstanding of the technology, though.
DiffusionGemma goes one step further even, and does this denoising over multiple "canvases" which lets it do reasoning and separate out a "final reply" canvas, looks something like this: https://gist.github.com/embedding-shapes/f4cb46bad704b6d0168...
Diffusion text models for me is the more interesting type of LLMs for local usage, as it really makes good use of single GPUs for single responses, rather than auto-regressive ones, and is a lot faster! Probably the fastest model I've been able to run so far, ending up doing ~670 tok/s (depending on the type of text) on a Pro 6000
Denoising is probably the weakest part of this model. There is recent research from Kaiming He showing that predicting the noise is actually not the best strategy for image generation since noise space is so large.
Simply predicting the surface of the data you are trying to generate is far more representationally efficient, and perhaps in the next few months we'll see a version of that for LLMs
Diffusion language models work with a discrete output space, unlike image models that repeatedly refine a continuous output, so they don't do the noise-prediction thing anyway.
Ok, you are correct. These models don't train noise predictors at all unlike first gen image diffusion
How does that break your brain? It's how basically every human writes and iterates on text..?
Because my brain thinks through text in a forward motion. Pausing at the end of each word and searching for the next.
My entire brain runs on sentences and words since I have no inner eye or whatever. So my thinking and writing both work kind of forward only.
I wouldn’t have thought that was too unique. But maybe it is?
Normally people have feelings about things before they are able to put them into words, I would imagine if you were asked a question like "what city would you most like to visit" then unless you've already thought about it a lot, then you would have to do substantial non-verbal thinking before you can come up with an answer, and once you have the answer you may respond "my favorite city is X" and you decided what X would be before you started the sentence.
Here's the best way I can explain how I "think" because it is mostly passive, but sometimes, I am actively thinking.
If I'm asked a hard question, like what city I'd most like to visit, I kind of have to look up cities and cross reference them with ones I've noted that I want to visit, or have visited and liked enough to go back, then actually read the list, and figure out which one, but each time I read a city, I might have a, this would be great in summer or winter or they have a cool festival in June. Which leads to additional question I have to ask. Which I think about roughly the same speed I speak, maybe a word or two faster.
My wife is the complete opposite. She can put together a sentence in an instant in "parallel" so when she says it, she though it already, but not as a sentence.
Human brains all work so crazily weird, and we are only just now started to realize all the nuances.
Does a human brain start with pure noise and then move that noise subtly towards the target?
From what I've noticed at least for myself, there are like at least two distinct parts that don't always cooperate. There's a draft part that's random thoughts that just appear and can be converted immediately to speech, but when writing it's actually another part that takes dictation from that one, albeit silently.
If I'm just typing blind I'll somehow end up writing random homophones down with completely correct spelling, like hear instead of here, it's bizarre. So that part is sort of structured autogenerated noise that is then consciously either appended or inserted into random spots in text. Maybe it's more of an LLM first pass then diffusion refinement.
Appealing results... do we think there is scope to close the accuracy gap against AR models? or even leverage the "Bidirectional Reasoning and Self-Correction" into an overall advantage?
there's still JEPA to be integrated before AGI.
Would DiffusionGemma be suitable candidate for DFlash 2?
DFlash 2 is a diffusion-based speculative decoding head for autoregressive models; there's nothing to accelerate here, because this is already wholly diffusion.
its more of a competitive approach to improve compute utilisation at lower batch sizes
(to finish your thought) Which is important for consumer hardware to be better suited to running these models. Cloud providers are already able to batch as many requests as they want together to improve resource utilization, so they will not see a big benefit from diffusion models.
Has JEPA shown a shred of viability yet?
Anybody was able to run this model in a server?
Yes, it runs in vllm happily. It gets >900TPS output reliably on a single 5090 with the nvfp4 model.
It's clearly worse than vanilla 26B-A4B, and lacks some things like structured outputs, and gets some tool calls wrong.
So you have to find a usecase or a hand rolled harness that leverages the cerebras-level TPS while not going off track during (even short) tasks.
And it fails on rocm of course. This engine is such a hassle on AMD.
That's AMD's fault.
RDNA4 is pretty similar to CDNA4, yet over a year after the release of "pro AI" cards like the r9700, they had basic kernels lacking in vllm (like w4a16 int4 kernels) while they were implemented in the datacenter CDNA4 cards.
AMD hardware runs well on llama.cpp because basically anything runs on llama.cpp, especially with vulkan. It's not high praise of AMD's software team to say llama.cpp runs well on their hardware