I maintain a fork of ds4 as shared libraries and thus can be used with other languages via FFI, along with public builds/binaries [1]. I made ds4go [2] against ds4 using techniques inspired by yzma.
In addition to the library bindings, we have a small library of tools (workspace for view/edit, scratchpad for persistence) and making your own is registering a Go function. And in recent weeks, I added the Vision and Qwen support, as ds4 added them.
Even if you don't use the Go library, the ds4go binary makes it really easy to download the libraries off of HuggingFace with a TUI available vie Homebrew.
Here's some TUI toy screenshots, sorry I still haven't released that code; it's of different quality than the others. [3]
Nothing comparable but inspired from DwarfStar I wrote a little inference engine for Intel Xe-LP (no XMX) 32GB laptops. The only model supported right now is a quantized Gemma-4, but I don't exclude in the future to support other MoE of similar size. Too bad we have no Qwen 3.8 35B-A3B yet.
I'm also looking into expanding the protocol and the engine to support various steering techniques.
Just tried this on my Intel Ultra 7 255H, I also only have an iGPU. This does ~22tps! Love this.
I just had to do a little patch to support my iGPU device that is a bit newer than Intel Xe-LP, maybe I'll do a PR.
On a side note the other day I was experimenting with Sonnet 5.5. I gave it the llama cpp repo and told it to extract in a single file inference for a single model + backend (qwen3.5 4b mtp + sycl) and (after a long time) it actually worked! It produced a ~1400 lines file with no deps. I need to check the quality of inference yet but I think this is still a great achievement.
I'm pretty sure 2027 will be a very interesting year for local models and inference.
Yeah I see the value but I built Xenolith to target smaller models.
I heard antirez saying that he designed DwarfStar also to be forked and tuned to everyone's specific needs. Do you have a specific machine/spec in mind?
Ive been using this since it was initially released with deepseek v4 flash, and it is absolutely the best launcher ever on my m5 max 128gb
Now Ive been running qwen 3.8 flash next for more than a week and it’s doing great, really fast and super long context windows. Sometimes the model is behaving stupidly by not remembering something I said earlier but it could be also a problem from the agentic AI harness. Im using oh my pi but Im wondering what people are using ds4 with here ?
It is pretty nifty. I spend some time over last weekend implementing fused TQ to allow for 1m context lengths on a 128 gb MacBook M5 Max when using Qwen 3.8 flash next (https://github.com/antirez/ds4/pull/1115 if you are interested). If I get bored I might port over the Metal kernels from oMLX -- the speed increase they have for the v0.7.0 release is amazeballs.
small native inference engine optimized first for DeepSeek V4 Flash (including the experimental vision model), DeepSeek V4.1 Flash (Metal, and text inference on CUDA), and additionally GLM 5.2 and 5.3, GLM 5.3 Flash and DeepSeek V4 PRO, and Qwen3.8 Flash Next (Metal and CUDA)
This is local targeting high end consumer hardware like DGX Spark or AMD Ryzen AI Halo.
For our mere mortals that were kids not long ago and can't really believe we've got our hands on a x090 series targeting Qwen3.8 27b, https://github.com/noonghunna/club-3090 is the way to go.
First of all, a x090 series card is "high end hardware" in its own right these days. Secondly, I'd think you'd probably get more interesting results running a MoE model in CPU-MoE mode, i.e. with the shared parameters residing on GPU and sparse experts on CPU plus SSD offload. Yes it will be slower, but small dense models are just a dead end and not that interesting. (Note that prefill would still be sped up in this setting; the CPU/GPU layer split in llama.cpp and the like applies to decode, but even a "0 graphic layers" setup does accelerate prefill.)
If you are interested in high-end models with high-end Apple Silicon, also try out Local Code: https://releases.drawthings.ai/p/public-beta-of-local-code-b... It is currently in TestFlight (and will open-source next week), supporting vision with DeepSeek 4.1 Flash, Qwen 3.8 27B and DeepSeek 4 Flash 0731 without vision. Custom quants & SSD streaming to make these big models work with 64GiB and above devices (and of course, Qwen works with devices with 16GiB and above).
Emphasis on performance and usable coding/agentic ability for consumer AI hardware. Does not attempt to handle all models or hardware at once but rather focuses on optimizing the best options for that category of hardware.
It's more likely to work. Most LLM runners are meant to work with any model, which means there are all kinds of ways you might misconfigure them in a way that causes function tooling not to work, or performance to be less than you would like.
DwarfStar's selling point is that it only supports a small set of carefully chosen models, but it supports them really well.
Lots of small details are taken care of so it runs smoothly. For example ds4-agent is append only, never rewriting history of messages, keeping KV cache prefix reusable. Huge benefit
Antirez has been writing C for a million years so is much more familiar with it than Rust.
Also the goal of the project is to squeeze the absolute maximum performance and capability possible out of limited hardware resources (compared to clusters of B200s or something).
Does Rust even give you good access to low-level code on different platforms? And if so, how much extra work do you need to do to make it acceptable to the compiler? And is that work worthwhile if you are not going to get the security guarantees of normal Rust code? Is it a worthwhile tradeoff when the goal is performance?
Those are real questions by the way, not rhetorical. If Rust could work well for this type of project then I would like to know.
Yes, Rust gives you great access to low-level code on different platforms, including SIMD. It is also alias-free by default, and gives you excellent primitives to write multi-threaded code with compile-time correctness guarantees, which is how projects such as zlib-rs end up significantly faster than their C counterparts.[1]
It's about as good as it can get for this kind of code.
> Antirez has been writing C for a million years so is much more familiar with it than Rust.
This is explicitly an AI-coded project, Antirez argues that LLMs are worse at writing Rust than C because so much high quality systems code (think e.g. sendmail) that ends up in AI training sets is C, not Rust. Another related argument is that the more detailed syntax and compiler feedback found in Rust compared to C are really a negative for LLM workflows.
There's plenty of room to disagree wrt. this of course: without the strong typing checks of Rust around e.g. indirect references, safety and correctness ends up being a global property in typical C programs, and LLMs are terrible wrt. reasoning about global properties. You're better off forcing them to adapt to a different local syntax that does a more complete job of enforcing modularity, since this is comparatively foolproof.
Personal preference of the author, he made at least one video on YouTube on why he dislikes Rust. I think he finds it too cumbersome and not worth it when the software isn't security-critical (not that I agree, just reporting what IIRC his stance is).
He recently said that he finds Rust less ergonomic and that this also affects code written by LLMs, which he thinks excel at writing C partly because of the enormous, high-quality codebase they were trained on. He sees security-critical code as a reason to choose Rust.
Right? Compare this with antirez's blog lmao. The very author of an incredible piece of software using the most plain website possible, while a derivative post about the same tool it's a slop fest with useless FX, cringe hackerman style palette and such. It's just too funny.
There are insane speed improvements for local inference going around on X right now. They've popped up the last month and week.
Tensorfold is getting 100%+ speed increases on both prefill and decode for models like Qwen 27B. oMLX has followed them and have had similar improvements in the past week.
There's lots of different techniques like letting CPU help with prefill, DFlash specualtive decoding etc.
I'm really excited for this as I'll be receiving an M5U in about a month. Expect to be running Qwen 4 27B or Flash (it's a 96gb machine), and they may come close in performance to DS 4/4.1 Flash, and should be able to hit 100 tps. Local is really becoming viable, especially considering that GPT 6.1 has been running at 20ish tps the past week.
I maintain a fork of ds4 as shared libraries and thus can be used with other languages via FFI, along with public builds/binaries [1]. I made ds4go [2] against ds4 using techniques inspired by yzma.
In addition to the library bindings, we have a small library of tools (workspace for view/edit, scratchpad for persistence) and making your own is registering a Go function. And in recent weeks, I added the Vision and Qwen support, as ds4 added them.
Even if you don't use the Go library, the ds4go binary makes it really easy to download the libraries off of HuggingFace with a TUI available vie Homebrew.
Here's some TUI toy screenshots, sorry I still haven't released that code; it's of different quality than the others. [3]
EDIT: add ds4go TUI screenshot gist [4]
[1] https://github.com/NimbleMarkets/ds4/releases/tag/v0.8.20260...
[2] https://github.com/nimblemarkets/ds4go#install
[3] https://gist.github.com/neomantra/ae47422c8daf7a458212c93992...
[4] https://gist.github.com/neomantra/40180ade13df93290250ce8c6d...
https://github.com/antirez/ds4
The project GitHub page is a much better introduction for the hn crowd.
Nothing comparable but inspired from DwarfStar I wrote a little inference engine for Intel Xe-LP (no XMX) 32GB laptops. The only model supported right now is a quantized Gemma-4, but I don't exclude in the future to support other MoE of similar size. Too bad we have no Qwen 3.8 35B-A3B yet.
I'm also looking into expanding the protocol and the engine to support various steering techniques.
https://github.com/simoneiacomino/xenolith
Just tried this on my Intel Ultra 7 255H, I also only have an iGPU. This does ~22tps! Love this.
I just had to do a little patch to support my iGPU device that is a bit newer than Intel Xe-LP, maybe I'll do a PR.
On a side note the other day I was experimenting with Sonnet 5.5. I gave it the llama cpp repo and told it to extract in a single file inference for a single model + backend (qwen3.5 4b mtp + sycl) and (after a long time) it actually worked! It produced a ~1400 lines file with no deps. I need to check the quality of inference yet but I think this is still a great achievement.
I'm pretty sure 2027 will be a very interesting year for local models and inference.
Please open a PR! I was too conservative with the supported devices.
If your GPU supports XMX we could also explore using it to improve the prefill kernel, but I don't have the hardware to test it myself.
I wish someone would add Intel support to ds4. And also improve AMD support.
Maybe Intel and AMD should help them with that.
Yeah I see the value but I built Xenolith to target smaller models.
I heard antirez saying that he designed DwarfStar also to be forked and tuned to everyone's specific needs. Do you have a specific machine/spec in mind?
The recent Intel GPU/AI cards. Really the same type of models as ds4
Wondering how well this does with tool calling. Any one has any numbers or videos or anything using this?
From the github repo it seems like you really don't need a big Mac with huge amounts of RAM but SSD is sufficient.
If this is anywhere near 50 TPS, that would be a game changer in the personal LLM space!
Ive been using this since it was initially released with deepseek v4 flash, and it is absolutely the best launcher ever on my m5 max 128gb
Now Ive been running qwen 3.8 flash next for more than a week and it’s doing great, really fast and super long context windows. Sometimes the model is behaving stupidly by not remembering something I said earlier but it could be also a problem from the agentic AI harness. Im using oh my pi but Im wondering what people are using ds4 with here ?
It is pretty nifty. I spend some time over last weekend implementing fused TQ to allow for 1m context lengths on a 128 gb MacBook M5 Max when using Qwen 3.8 flash next (https://github.com/antirez/ds4/pull/1115 if you are interested). If I get bored I might port over the Metal kernels from oMLX -- the speed increase they have for the v0.7.0 release is amazeballs.
I’m already able to use 1M context windows with the same machine than you and same model. Strange
Yeah, most of what I did was to add fused TQ support to leave more memory free for other nefarious purposes.
For our mere mortals that were kids not long ago and can't really believe we've got our hands on a x090 series targeting Qwen3.8 27b, https://github.com/noonghunna/club-3090 is the way to go.
I'm maintaining a web frontend for this, trying to at least. You can follow it here: https://github.com/gchamon/club-3090-server
First of all, a x090 series card is "high end hardware" in its own right these days. Secondly, I'd think you'd probably get more interesting results running a MoE model in CPU-MoE mode, i.e. with the shared parameters residing on GPU and sparse experts on CPU plus SSD offload. Yes it will be slower, but small dense models are just a dead end and not that interesting. (Note that prefill would still be sped up in this setting; the CPU/GPU layer split in llama.cpp and the like applies to decode, but even a "0 graphic layers" setup does accelerate prefill.)
Qwen3.8 35ba3b is decidedly faster but also dumber, at least in my tests.
I’m a little confused
ds4 is referring to “dwarfstar” “4” and references DeepSeek V4 most of the time
but its model agnostic-ish
and benchmarks compared to what? what do these large MoE models typically get in tokens per second?
I’m garnering this is just an easier way to load large models per expert on consumer hardware? as opposed to the hackier solutions?
I’m intruiged. Note that the blogpost says 64gb Macs are good minimums while the github says 96gb is a minimum
If you are interested in high-end models with high-end Apple Silicon, also try out Local Code: https://releases.drawthings.ai/p/public-beta-of-local-code-b... It is currently in TestFlight (and will open-source next week), supporting vision with DeepSeek 4.1 Flash, Qwen 3.8 27B and DeepSeek 4 Flash 0731 without vision. Custom quants & SSD streaming to make these big models work with 64GiB and above devices (and of course, Qwen works with devices with 16GiB and above).
How is this different from other LLM runners?
Emphasis on performance and usable coding/agentic ability for consumer AI hardware. Does not attempt to handle all models or hardware at once but rather focuses on optimizing the best options for that category of hardware.
It's more likely to work. Most LLM runners are meant to work with any model, which means there are all kinds of ways you might misconfigure them in a way that causes function tooling not to work, or performance to be less than you would like.
DwarfStar's selling point is that it only supports a small set of carefully chosen models, but it supports them really well.
Lots of small details are taken care of so it runs smoothly. For example ds4-agent is append only, never rewriting history of messages, keeping KV cache prefix reusable. Huge benefit
My instinctive reaction from the readme is that it isnt. It's apparently a vibe coded knock off of llama.CPP.
curious, why did antirez go with C instead of something like Rust ?
Antirez has been writing C for a million years so is much more familiar with it than Rust.
Also the goal of the project is to squeeze the absolute maximum performance and capability possible out of limited hardware resources (compared to clusters of B200s or something).
Does Rust even give you good access to low-level code on different platforms? And if so, how much extra work do you need to do to make it acceptable to the compiler? And is that work worthwhile if you are not going to get the security guarantees of normal Rust code? Is it a worthwhile tradeoff when the goal is performance?
Those are real questions by the way, not rhetorical. If Rust could work well for this type of project then I would like to know.
Yes, Rust gives you great access to low-level code on different platforms, including SIMD. It is also alias-free by default, and gives you excellent primitives to write multi-threaded code with compile-time correctness guarantees, which is how projects such as zlib-rs end up significantly faster than their C counterparts.[1]
It's about as good as it can get for this kind of code.
[1] https://www.reddit.com/r/rust/comments/1ixt1ei/zlibrs_is_fas...
> Antirez has been writing C for a million years so is much more familiar with it than Rust.
This is explicitly an AI-coded project, Antirez argues that LLMs are worse at writing Rust than C because so much high quality systems code (think e.g. sendmail) that ends up in AI training sets is C, not Rust. Another related argument is that the more detailed syntax and compiler feedback found in Rust compared to C are really a negative for LLM workflows.
There's plenty of room to disagree wrt. this of course: without the strong typing checks of Rust around e.g. indirect references, safety and correctness ends up being a global property in typical C programs, and LLMs are terrible wrt. reasoning about global properties. You're better off forcing them to adapt to a different local syntax that does a more complete job of enforcing modularity, since this is comparatively foolproof.
Yes, it is AI-coded. But definitely not a one shot kind of a deal.
Much easier to work with a language you are most comfortable with right?
Personal preference of the author, he made at least one video on YouTube on why he dislikes Rust. I think he finds it too cumbersome and not worth it when the software isn't security-critical (not that I agree, just reporting what IIRC his stance is).
He recently said that he finds Rust less ergonomic and that this also affects code written by LLMs, which he thinks excel at writing C partly because of the enormous, high-quality codebase they were trained on. He sees security-critical code as a reason to choose Rust.
The video is in Italian but has an auto-dubbed English audio track: https://www.youtube.com/watch?v=sOt0WpQG5eU\&t=526s
This website is pure slop. I'd ask @dang to just link the original repo
Right? Compare this with antirez's blog lmao. The very author of an incredible piece of software using the most plain website possible, while a derivative post about the same tool it's a slop fest with useless FX, cringe hackerman style palette and such. It's just too funny.
the problem is the dsv4 checkpoint so quantized isn't very good
Which ds4 checkpoint for which model exactly did you test? Don't they have multiple different versions and quantization levels?
not my experience
the ds4 quants were very good beating the unsloth quants https://github.com/michaelasper/benchmarks/blob/main/deepsee...
That's quite the statement - unsloth quants are amazing.
There are insane speed improvements for local inference going around on X right now. They've popped up the last month and week.
Tensorfold is getting 100%+ speed increases on both prefill and decode for models like Qwen 27B. oMLX has followed them and have had similar improvements in the past week.
There's lots of different techniques like letting CPU help with prefill, DFlash specualtive decoding etc.
I'm really excited for this as I'll be receiving an M5U in about a month. Expect to be running Qwen 4 27B or Flash (it's a 96gb machine), and they may come close in performance to DS 4/4.1 Flash, and should be able to hit 100 tps. Local is really becoming viable, especially considering that GPT 6.1 has been running at 20ish tps the past week.
Random comment but the name is funny to me, reminds me of Silicon Valley.
What are we going to name the company, how about Dwarfism 2.0? What happened to 1.0 Jared?
I immediately thought of the unique ring from Diablo 2 named Dwarf Star.
Dwarf Star is better than Dirty Socks, or Dynamic Slinky.. definitely not the worst backronym.
Smallulator