After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way.
I have almost completely stopped writing real in my own output as a direct consequence of this. Maybe next year they'll catch up with my alternative wordings and I'll end up switching back
That was their headline for Opus 4.8, I guess the invested into some post-training to get them to write this, and I like it, it's a great way to identify AI posts from Claude.
In the future, I expect different models from different firms behaving differently will become as obviously normal as different humans behaving differently. Those of us that have used agents a lot have already noticed this, but the general public still seems to consider AI to be plug-and-play.
The Opus 4.8 infatuation with being honest genuinely drove me nuts (honest take). The constant need for me to decipher and decide on something after its final message was tiring too.
GPT-5.6-Sol is refreshing that its replies start with "understood." instead of flattering me for my steering prompt. Now ... after a few weeks, will I get suck of "understood"? I dunno :)
It's as if it's sucking up to me or apologizing at every turn, even when it's done exactly what I've asked. Seems related to sycophancy, but different? (i.e. rather than unconditionally affirming the user it's unconditionally expecting to disappoint)
My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".
For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise.
Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
Which CPU gen are you suggesting, is there any writeup on such setup where <10K (not incl. power bill) cpu only rig is giving usable token speeds on latest SoTA open weights models?
In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment.
Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control.
Kudos to OP though, They had clear goals and they achieved it.
I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB) and then adding the DIMM kits from the more reputable refurb server part vendors on ebay.
It won't be fast at all, for certain, but it'll have enough memory to prove a configuration and be able to really use gargantuan GGUF format LLMs in the latest compiled llama-server. Re: electricity, I pay the equivalent of $0.07 ro $0.09 USD per kWh so it's not an extreme burden to have a theoretical 500W server running. Something like $35 to $50 of electricity a month if it's 500W 24x7.
Xeon Scalable in general seems like a good idea due to 6-channel (relatively) inexpensive RDIMM memory, but I've been reading that NUMA kills inference performance. Anyone got experience with multi-socket systems? IIRC even within the socket these cpus are divided into sub-numa nodes.
Even though LLM benchmarks are very opinionated, I would really like to see some numbers for the setup parent suggested. From what I read elsewhere, anything below $40K in HW costs is not worth the effort for coding models locally.
The old Cascade Lake based server found by the previous poster is still new enough to have instructions for relatively fast AI inference with the INT8 format.
So for optimal speed the models must be quantized in this format.
It is very likely that with INT8 models those CPUs are fast enough so that the inference throughput is limited by the memory bandwidth (384-bit interface to DDR4-2933 per socket, i.e. 282 GB/s for both sockets).
The memory throughput for such an old server is very similar to an AMD Ryzen Strix Halo, NVIDIA DGX Spark or Apple M5 Pro, but it has much more memory.
The inference speed should be very similar to those, but with bigger LLMs.
Would be nice if you could somehow connect GPU-levels of parallel floating point cores to that amount of memory. I guess that's what the big AI datacenters are doing, but how can we do that on a budget?
I think there is a good sized population of people who absolutely don't want to submit everything they do to an off site service, or let their content be used for unknown training purposes, and will tolerate slowness at 1 to 10 tok/s as a tradeoff.
Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.
Not just controversial but also regulated areas. Virtually every law firm would be interested on locally-hosted AI at a reasonable price. So too ever medical research lab. Every CGI firm doing work for film/TV. And all the video game developers.
Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...
For one, doing the math on what it costs to rent a 768GB+ RAM AWS system with 40+ high performance CPU cores makes it very unappealing to pay for 12, 24, 36 months of it.
The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload.
The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly. I just did the math on one of these and it costs $12 per hour, or $289 a day, or $8900+ for one month.
Also plenty of Europeans or people from other locations may consider it as an unacceptable risk factor to put their "off site" self hosted AI stuff with an American controlled company. Particularly if the servers are physically in the USA.
Hetzner will also rent you 768 GB of RAM with a Blackwell 6000 Max Q GPU for €2300/month [1].
Yes, it's a boatload of cash, but that's a €13,000 GPU and €20,000 of RAM at present prices. There is a segment of businesses where a fixed €28k/year bill is going to be preferred over plonking down €40k for a (theoretically) depreciating asset and ongoing colocation costs.
Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me. And yeah it depreciates, but not to zero. So if you're speaking of the breakeven point after liquidation, you're probably there in well under a year at those rental prices.
> Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me
And yet basically all AWS customers are doing exactly that. Turns out that making CAPEX "someone else's problem" is worth quite a lot to many businesses
that would be implying that "private" really means anything for AWS. Because if it's "private" as in "private" github repos that were totally not used for training copilot because they said so or "private" claude chats that are totally scanned even if you have enterprise contracts to check you are not doing anything malicious or are from china or whatever, and this will totally not be used for training...
can we trust any US based service to guarantee privacy and confidentiality? especially to us european frienemies?
> that would be implying that "private" really means anything for AWS
Insert your dedicated hosting provider of choice for 'AWS' (somewhere like Hetzner will be cheaper anyway).
But in general, AWS hosts are yours, running your code, with your security policies enforced. Sure, the US government can silently subpoena the contents thereof, but aside from that fairly extreme case, it's not like AWS is handing your data over to 3rd parties.
I would suspect that one would buy based on mem-bus & PCIe bus speeds more than CPU for this, and just dial down the CPU parameters to save power. Most of the time and power will be consumed by memory and bus transfers because the CPU will mostly be waiting to the right set of weights and factors to multiply.
Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.
This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
Could you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably.
Certainly. First of all, I am using OpenCode as the harness. (I have heard there are better harnesses such as little-coder for small open-weights models, but I haven't tried them yet.) Looking over some of my recent sessions, here are some examples:
- Asking Qwen to review project docs (requirements, user stories, etc) so that "we" can evaluate an iterate on an API design. Then back-and-forth chat about possible design directions. Then I ask for a rough-sketch plan of the one I'm interested in. I provide some tweaks to the plan and request a final plan in full detail. I switch to build mode and say go; everything is written to spec.
- Asking Qwen to write a suite of tests covering X, Y, Z issues with permutations A, B, C per issue.
- Asking Qwen to edit the shape of a CNN to insert auxiliary branches for intermediate supervision, and to extract out part of the network as a modular component with parameterized architecture.
I have less experience with the dense 27B because it's too slow to use on Apple Silicon. But regardless of which model you try, I would recommend trying a full-fat cloud hosted version of it first, so that you can get a sense of what it's capable of when the inference stack is correctly configured. LLMs are very sensitive to quantization formats, discrepancies in chat templates, etc. That kind of stuff is make-or-break.
The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken"
Caveat: I have not been able to try that model locally, so no personal experience. Running this locally at usable speeds would be cost prohibitive for personal coding use for me.
But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
> OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
While it probably won't matter enough to change your mind, remember that you've gotten better at extracting value from all models than you were a year ago - plus the harnesses and other tools have gotten a lot better too.
Knowing what to ask for, for one. Nobody can just whip up a specification for a system that satisfies all of the technical/design/business constraints that will turn out to have been relevant, has good usability for the target users, hits the right performance tradeoffs - all out of thin air. If anyone could, THAT would be priceless.
Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028.
Expectations seem to be rising at a faster rate than models can improve.
I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.
Having a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make.
The hard thing is always keeping complexity low and being ZeroOps.
Most any ticketing system can integrate with ordinary IMAP and smtp email flow, so you can really use any agent that can "do" inbound and outbound email to talk to a self hosted ticket queue.
So I’ve been thinking about this problem a lot, specifically as it relates to running LLMs at home, and I’ve been using GLM-5.2 to make an SMTP/IMAP-to-LLM gateway.
I’ve been wondering about something similar - a system that enforces (or does the heavy lifting) of dividing a large task into smaller sub-tasks so that it’s easy to run/check/test each one independently - even on a fresh model instance if needed.
This is based on the observation that the medium-sized open weight models (~20-35b) are very able to one-shot smaller discrete tasks but seem to lose their way project managing themselves through larger tasks that have multiple steps.
Now many mini-PCs and desktops are able to read simultaneously from 1 PCIe 5.0 SSD and 1 PCIe 4.0 SSD. This can ensure a reading throughput around 20 GB/s, i.e. 20 times faster than on author's system.
With only 1 PCIe 5.0 SSD, the reading throughput is still significantly more than 10 times faster than on author's system.
So it is likely that inference speeds around 1 token/s are achievable on something like a NUC mini-PC.
In the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve!
I have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).
If you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself.
On a level playing field the expression of virtuosity can outshine those who have never known any instrumental limitations at all :)
When pulling way more than your own weight happens like for few others.
There should be an award for getting the most out of the electronics rather than trying to reach orbit by building the tallest pile of e-waste.
First Prize right before your eyes !
Grande praise !
And just starting to ascend toward an unconquered summit that others find forbidding ;) Or they find uninteresting since the limit naturally lies on firm earth somewhere below the stratosphere.
0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a sizeable fraction of the complete model at every token batch in order to get good reuse) and is ultimately limited by CPU/GPU thermals which are a tight constraint on typical inference platforms. It's also only really feasible with tiny KV caches, which requires either a very small context or sticking to KV-cache efficient models such as the DeepSeek V4 series. Still, this might be one way of making use of existing lower-end hardware for practical inference of non-tiny models.
For most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage.
I bought whole Intel N100 mini pc with 16GB of DDR5 in it in 2023 for $AUD289 (so about $US200). I got a 16GB (DDR4) SODIMM in 2022 for $AUD88 ($US60).
I was just using that as an example of constant on going price rises, it was the most mundane and not particularly fast ddr5 6000 stuff. The 6400 is even more ridiculous.
Maybe that's a measure of the self-fulfilling dollar incentive toward "renting" someone else's RAM in the future rather than trying to actually own such an outlandishly luxury item :\
Ideally this engineer's approach will yield better performance on lesser equipment in the future, if they keep up the good work after they get more-capable gear to experiment with as time goes by :)
Working on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.
I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet.
To expand since I just got home, I'm making all of my modifications to llama.cpp, the goal was to eventually put this on a SBC of some kind with an nvme to handle the mmapped files. I think the theoretical limit of my current setup is about 1.8 tok/s based on prior testing but that is also with the additional medusa heads not fully trained (I honestly don't know if the counting it's generated tokens or not.)
In the end it seems like the idea we had is similar, I just don't know how to write an llm parser/runner from scratch yet and instead of specifying what needed to stay in memory I just let the linux kernel handle it.
Oh last note, I also capped llama.cpp usage to 16GB of my 32GB, so it might be possible to get it down even lower.
Not sure if mmapping is the right way. In my own tests I noticed that simple mmapping will produce many small reads and not keep the SSD queue saturated. So if RAM is large enough to cache most experts, that is a rounding error. But if the base weigths without experts fill more than half or RAM and you basically need to load in a few experts for each layer of each token, the latency gets important and mmap sadly blocks until the data is loaded. You can't do concurrent requests for multiple experts with mmap (but you know all the ones you need right after the router ran). And even going one step further, depending on the arch / the tensors you need, you could eagerly load some, start computing with them and load the rest of the expert tensors in the background (extra thread or async io) parallel to the compute. This is not really possible with mmap, even with madvise.
One further step is predicting which experts will be needed next token / next layer. LRU does this okish. But a learned projection from the hidden state can do better. Or even a simple correlation from past activated experts. Expert usage is heavily skewed.
On the technical details of mmap I agree (at least while single threaded which is how I believe I'm running it), but making it async does sound like an interesting method for speeding it up. My only goal with my testing was get as large of a model as possible running on a computer that "can't run it." I'm going to have to actually read through the code and figure out how it really works to really make any good optimizations, but as before this was just experiments with using and LLM (claude + codex) to just get it running. Since if they couldn't get it running I'm not sure if I would have wanted to spend time trying to get it working myself.
I also know I did some things that would actually make the perf worse to, like I believe I also had AI mmap the KV Cache to make sure to runs under any circumstance. For actual optimizations based on what I currently know, I'm probably going to try and get the llm running under my igpu on my laptop with persistent shader that has some kind of inbuilt request mechanism. That way the weights that are loaded can be used as fast as possible.
For the expert prediction, I assume I could use the medusa paper as kind of a kick off point for that since I'm already using it to try and predict the next 4 tokens. Doing verification on those 4 tokens is about as much as I can do though since it started to thrash on loading the experts. So some method of predicting even more tokens, but then batching together those with the same experts would probably yield slightly better results in this weird case.
Note: All of my tests have been around programming since that's the use case I'm interested in. I don't actually know if this would preform well in other cases (and anything more broad than that I assume would be slower.)
Yeah I'll see what I can transfer over from my llama.cpp work. As before I'm not too experienced with llm work, but I have a lot of experiments I'm trying out. So I'll make a PR if I get any interesting results.
Let me know if you want to hear anything specific about it. It kind of works, so it's not something I recommend doing if it can be avoided, but as Roxxik pointed out there is much room for improvement since this was just a naive just get it to run experiment.
llama.cpp supports a wide variety of 4-bit and smaller quants and mmap's models by default, so you dont need to be able to hold the weights in memory (the OS will handle bringing them in from storage as needed)
Its cool to see this implemented in a tiny amount of code without dependencies, but does it actually bring more performance?
Basically I kept needing an inference engine that could stream weights in and out as needed in an LRU manner. So I ended up vibe coding this thing that accepts a `--vram-budget` and stays under it (mostly). It turns out moving mmap'd bytes in and out of VRAM is way cheap compared to compute. Coupled with some pipelining/double-buffering, I almost always end up compute bound not memory bound. Granted I use way smaller models heh.
Wow, I see you managed to fit in so many models (krea, wan, hunyan, etc.). Did you get to build a common harness to run all of them? Which ones stay under your VRAM budget more consistently?
All stay under because I had Claude build the workflow to respect it (text encoding, denoising, vae, etc), there's just a tiny bit of untracked pieces. While there are common interfaces to invoke them (CLI and API/webpage) and they share ops and some pieces, lots of model logic is unique. This is all vibe coded and surely has inaccuracies.
Looking at the progress with DS4 project from Antirez and Colibri, are we probably going in the direction of having a stripe RAID of SSDs instead of RAM to get decent performance without losing a kidney to the RAM mafia? I really hope so.
Pretty cool! I've also been playing around with GLM 5.2 this week and was equally impressed. At work we're running it locally on some crazy expensive hardware as a test before starting another project so it's great to see people taking this massive FOSS model release and running it on an average machine, even if it's not terribly practical at this point.
I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2
I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.
>SSD Wear Warning
> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.
Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only?
Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?
I have a small laptop.
If you have more disks available, you could really do some testing.
When you have some benchmarks, submit a pull request or issue so we can maybe work on them.
We are really happy for contribute!
I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused.
I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU.
If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item.
I'm going to benchmark this right now with what I have and I'll get back to you on github.
> Is this a hallucination? What am I missing? Why would heavy reads generate writes?
I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?
I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?
The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD.
It's a very conservative warning. The application does not perform writes, so the application doesn't actually wear your SSD at all. The rest is just application-independent general hygiene.
Even under AppleCare this is a $400 service which for an older macbook costs almost as much as the whole thing. And without Applecare it's not worth fixing at all.
I’m truly impressed by your work !
I don’t know if this is planned for near future, but how about adding energy efficiency benchmarks ?
Because running locally is a great feeling, but the electricity bill should not be forgotten
I'm curious but don't know much about the internals of LLMs - could you use a similar architecture with other models that have "layers"? I mean, could you have one layer do its work, then remove that layer from RAM, load the next layer from disk, and have that layer activate on the result of the first layer?
I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.
It's been interesting learning about the balance of factors for performant metal kernels on unified memory.
Should have a repo up on github in the next few weeks.
I'd be interested in seeing this when you put it up! I've managed to get upwards of 300 tokens a second somewhat consistently on the desktop I already had gotten for gaming/personal project stuff several years ago (64 GB RAM, Ryzen 9 7900 X 12-core; I already had a fairly large m.2 SSD and a Radeon 6900 XT to plug into it), but only by severely compromising on the model (a custom quantized Qwen3.5-27B-Q2_K which someone published on huggingface) so that I can use a context size large enough that interfacing through opencode doesn't manage to run out of context while summarizing for compaction and then trigger a second "nested" compaction that instead just seems to cause it to lose all context and ask for a full new set of instructions from scratch. If yours ends up being anywhere as close to the one I've been using in quality, and it works on a 36 GB Mac, it sounds like it would be worth it for me to try out!
(For clarity, I have almost no idea what pretty much any of the terminology for models even means or how it translates to what the actual experience is. My strategy has mostly been using the free tier of OpenCode Zen to ask it stuff about what models and llama.cpp configurations to use in the hopes that I could maybe bootstrap something halfway decent to use locally, and so far what I've managed to get running is mediocre but at least passable)
I'm not fully understanding this business of MoE so please forgive me if this is a dumb question, but would it be possible to use MPI with a small cluster to distribute the load?
This is great stuff, I feel like fast disk (SSD) is a somewhat solvable problem, if you have many disks with the same content and a fast controller (?).
I you look at https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token. The experts aren't divided along predictable lines.
I have a similar question and I’m inferring the answer is no - look at the cache hit rate of 23% for the 128GB M5 Max. I had previously assumed that the 40B active meant that a set of layers was chosen as THE expert for a given prompt and generation was then limited to those layers until complete. But in that case you’d have expected the expert caching to have a super high hit rate once you had enough RAM to hold an entire expert’s worth of layers.
You could (e.g. by replacing residual-dependent expert routing with hardcoded logic), but quality will suffer dramatically. It’s far better to use a similar-sized dense model then.
I just learned about Gemma4.pas at the beginning of this week. Now this. This make me wonder how can inference engines could be built that easy. I'm not knowledgeable in this, but I thought it would take very deep Mathematic and system level knowledge, ... and a lot of patience.
If I understand correctly the basic structure hasn't changed since Transformers were invented. It's just gotten bigger and the models have had better training and also some optimizations. I don't think this one has optimizations.
the math involved is not very hard to understand. it’s linear algebra. the transformer model is brilliant but simple, nobody even really realized the impact it would have until they started training it on massive datasets
NVMe SSD prices had being gone down in price for a while, and the spikes are actually a lot more recent than you might think. From double checking my Amazon history, I bought my wife a 2 TB NVMe SSD for $160 back in November; it's now listed at three times that. I imagine that a lot of people just have them already from the past few years.
You've basically build an LRU page cache with readahead for streaming 21, 504 experts off disk.
Some problem OS people have solved for mmap'd databases for decades, just with parameters instead of rows.
I think if you had something like a theoretical used/refurb 2U rackmount server with two older multi core CPUs, 768GB of RAM, you would see faster performance loading a Q6 or Q8 GGUF of GLM5.2 into a freshly-compiled latest copy of llama-server, with the "no-mmap" option turned on to intentionally load the whole thing into RAM at the time the llama-server daemon launches.
If you want a CPU-only machine with 512GB to 1024GB of RAM, despite extreme cost rises, there are still some great options out there from companies selling ex-lease stuff that's 3, 4, 5 years old. It'll be loud as hell under full CPU load when running inference, so if you plan to use it at home, put it in your garage or basement or laundry room or somewhere similar on the far end of a network cable.
The software that OP has published appears to be specifically designed to hold only the active parameters in RAM (<100GB) and read content off local NVME SSD as needed on the fly. All that NVME SSD read wouldn't be necessary if you can hold the model in RAM, even in the absence of any GPUs.
AMD Ryzen Threadripper PRO 5975WX — 32 cores / 64 threads, Zen3 (znver3), AVX2+FMA (no AVX-512/VNNI), 128GB RAM, Kingston SKC3000D 4TB NVMe (PCIe4). Disk gets around 7GB/s. It took a little tuning (for example pinning to 32 physical cores instead of the 64 threads), but with that and --topp 0.7, got 0.44 tok/s on a cold start. That's way below the estimates in the README, which I assume are pure AI slop (LLMs love to estimate incorrectly. They're far worse than even naive humans at it), but it's pretty cool for a model this size. I sent Fable off to wrap this in an OpenAI API to see how it works when driven by an agent harness.
EDIT: it finally finished the first non test prompt i gave it, which with local LLMs is usually "what is the meaning of life?" (who knows, maybe one of them will finally answer). It got stuck in a loop, which is not encouraging, so there's a lot of work to do to make this a viable local coding tool:
> The meaning of life is one of the oldest and greatest questions in human history, yet strangely, there is no single, universally agreed-upon answer. Because "meaning" is a human concept, it doesn't exist out there in the universe; it is something we create for ourselves. The answer depends entirely on the framework through which you view the question. Here are the most common ways to answer it. The meaning of life is the meaning you give to it. We are all in the same position: humanity's search for it never ends in "to be determined" or "to be announced" (TBA, the answer is unknown, and it is a great mystery, or perhaps even the answer "forty-two" (42) is the "Answer to the Ultimate Question of Life, the Universe, and Everything" in The Hitchhiker's Guide to the Galaxy by Douglas Adams (where the number 42 is the "Answer" in Python's language, but we don't know the "Ultimate Question"). Here is a joke that works under the frame of "A..." (any answer): "A clean desk is a..." (42 is a "portmanteau" of words and just a great big "Ad..." (Ad-100) and "A&d" (100)). Life is a deep and strange and we search for meaning in it. "I think, therefore,..." (Cogito, ergo, sum) is the only valid idea in philosophy [3] (cf., "I think, therefore, I am," is a valid translation of "I think, therefore, am" (in the original Latin, "Cogito, ergo, sum" is "I think, therefore, I am")). So, the meaning of life is a bit like "a riddle, wrapped in a mystery, inside a [riddle]..." (G. K. Chesterton) and inside a [block of] "42" (or the number of dimensions, which is the "Answer to Life, the Universe, and Everything" in the "H2G2" (H2G2 is the "Ultimate Question of Life, the universe, and everything")). The "H2G2" is a "puzzle, wrapped in a mystery, inside an enigma" (cf. [3]). We are all in the same position, but we all have to give it a meaning. our own meaning. The meaning of life is what you make of it. The meaning of life is to live for the greater good. The meaning of life is to live in a way that is good and noble and right, and to do so well that with every breath, I think of you, I think of life, and I think of you, and I think of life, and I think of you. (cf. [3]) If life in the universe is a "great question," the answer is 42. The meaning of life is the meaning you make it. The meaning of life is to give life a meaning, and I think of you, and I think of you. So, the answer to the ultimate question of life, the universe, and everything is: 42. The meaning of life is 42. The meaning of life is the meaning of life. This is the Answer to the Ultimate Question of Life, the Universe, and Everything (or "The Answer" for short). It is the Answer to "the" Ultimate Question of Life, the Universe, and Everything. (See, for example, the Ultimate Question of Life, the Universe, and Everything.) This is the answer to the Ultimate Question. This is the Answer. (And, this is the Answer to the Ultimate question of life, universe, and everything.) The meaning of life is the meaning you give to it. The meaning of life is to give it a meaning. The meaning of life is the meaning you give it. The meaning of life is the meaning of life. The meaning of life is the meaning of life. The meaning of life is the meaning of life. (This is a list of the possible meanings of the universe of life. It's a list of the most common and accepted answers. "What is the meaning of life?" The answer is 42. The meaning of life is the meaning of life. The answer is 42.) (See also: [3] for a list of possible meanings.) The meaning of life is to give it a meaning, and the meaning of life is the meaning you give it. The meaning of life is the meaning of life. The meaning of life is the meaning of life. The meaning of life is the meaning of life. (This is the answer to the Ultimate Question of life, the universe, and everything.) (This is the answer to the Ultimate Question.) The meaning of life is 42. The meaning of life is 42. The meaning of life is 42. (See also: [3]) (The answer to the Ultimat
I’ve been looking at exactly this kind of system (in a Lenovo P620) to fill with external GPUs (powered externally and ribbon cabled into the pcie slots). What would you say was your best performing model on this system? And do you get any useful work done with it or are you still dependent on SOTA models online?
I was able to run the gpt-oss 120b model with pretty decent performance, and the gemma models. I haven't experimented yet with qwen3 on here much. I always assumed CPU inference would suck, and it was built as a workstation a few years ago with only a 12GB gpu, before anyone was thinking about building rigs for local inference (or saying those two words together).
What causes problems is the rewriting in this case are only read while writing is the cache! However, I'm working to improve more and more and make some parts lighter!
Another recent project that runs a huge model on a 48gb Mac is https://github.com/danveloper/flash-moe - it gets over 5 tokens/sec on an M3 Max compared to this projects very impressive 1 token/sec on an M5 Max. So for anyone wanting to tackle a Mac only version that targets lower spec machines this looks like a good candidate with plenty of room for speedups [edit: because it doesn't use the gpu].
Not hijacking anything as this project is amazing.
Assuming steady 1 tok/second generation (which seems to be the case for M5 Max macbook), wait 1 day for a 86400 token response. In some configurations it can be as slow as 0.1 tok/s, so be prepared to wait for 10 days.
With so many people implementing their own SSD streaming for specific combinations of model+hardware, maybe we should look into upstreaming to antirez/ds4 or llama.cpp...
After spending way too much time with Fable a few days ago, I noticed a new hallmark of AI generated text is using the word "honest" everywhere, in a somewhat self congratulating way.
Some things that give it away to me:
- "Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)"
- "an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires."
- "Honest caveat from the same measurement: ..."
It's the new "It's not X, it's Y". I have no issue with this, I just found it amusing.
Cool project btw!
I've been seeing a lot of usage of the word `real`
from recent fable sessions:
- That gap is the real story: 3,873 flows
- the real conversion filter types are:
- I'll update the breakdown query to include a column for each real type
I have almost completely stopped writing real in my own output as a direct consequence of this. Maybe next year they'll catch up with my alternative wordings and I'll end up switching back
Unfortunately we're much too late for realnetworks and realplayer to be relevant anymore.
I suspect it's a signature of anti-fabrication training.
That was their headline for Opus 4.8, I guess the invested into some post-training to get them to write this, and I like it, it's a great way to identify AI posts from Claude.
https://www.anthropic.com/news/claude-opus-4-8
Yup, and I had seen it in the Opus 4.5 soul document as well: https://gist.github.com/Richard-Weiss/efe157692991535403bd7e...
In the future, I expect different models from different firms behaving differently will become as obviously normal as different humans behaving differently. Those of us that have used agents a lot have already noticed this, but the general public still seems to consider AI to be plug-and-play.
The Opus 4.8 infatuation with being honest genuinely drove me nuts (honest take). The constant need for me to decipher and decide on something after its final message was tiring too.
GPT-5.6-Sol is refreshing that its replies start with "understood." instead of flattering me for my steering prompt. Now ... after a few weeks, will I get suck of "understood"? I dunno :)
It's not goblins, it's honest!
Claude is listed as a contributor right there
It's as if it's sucking up to me or apologizing at every turn, even when it's done exactly what I've asked. Seems related to sycophancy, but different? (i.e. rather than unconditionally affirming the user it's unconditionally expecting to disappoint)
My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".
The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028.
(I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise.
Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
Which CPU gen are you suggesting, is there any writeup on such setup where <10K (not incl. power bill) cpu only rig is giving usable token speeds on latest SoTA open weights models?
In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment.
Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control.
Kudos to OP though, They had clear goals and they achieved it.
I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB) and then adding the DIMM kits from the more reputable refurb server part vendors on ebay.
It won't be fast at all, for certain, but it'll have enough memory to prove a configuration and be able to really use gargantuan GGUF format LLMs in the latest compiled llama-server. Re: electricity, I pay the equivalent of $0.07 ro $0.09 USD per kWh so it's not an extreme burden to have a theoretical 500W server running. Something like $35 to $50 of electricity a month if it's 500W 24x7.
Xeon Scalable in general seems like a good idea due to 6-channel (relatively) inexpensive RDIMM memory, but I've been reading that NUMA kills inference performance. Anyone got experience with multi-socket systems? IIRC even within the socket these cpus are divided into sub-numa nodes.
Even though LLM benchmarks are very opinionated, I would really like to see some numbers for the setup parent suggested. From what I read elsewhere, anything below $40K in HW costs is not worth the effort for coding models locally.
The old Cascade Lake based server found by the previous poster is still new enough to have instructions for relatively fast AI inference with the INT8 format.
So for optimal speed the models must be quantized in this format.
It is very likely that with INT8 models those CPUs are fast enough so that the inference throughput is limited by the memory bandwidth (384-bit interface to DDR4-2933 per socket, i.e. 282 GB/s for both sockets).
The memory throughput for such an old server is very similar to an AMD Ryzen Strix Halo, NVIDIA DGX Spark or Apple M5 Pro, but it has much more memory.
The inference speed should be very similar to those, but with bigger LLMs.
Would be nice if you could somehow connect GPU-levels of parallel floating point cores to that amount of memory. I guess that's what the big AI datacenters are doing, but how can we do that on a budget?
I think there is a good sized population of people who absolutely don't want to submit everything they do to an off site service, or let their content be used for unknown training purposes, and will tolerate slowness at 1 to 10 tok/s as a tradeoff.
Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.
Not just controversial but also regulated areas. Virtually every law firm would be interested on locally-hosted AI at a reasonable price. So too ever medical research lab. Every CGI firm doing work for film/TV. And all the video game developers.
Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...
For one, doing the math on what it costs to rent a 768GB+ RAM AWS system with 40+ high performance CPU cores makes it very unappealing to pay for 12, 24, 36 months of it.
The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload.
The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly. I just did the math on one of these and it costs $12 per hour, or $289 a day, or $8900+ for one month.
Also plenty of Europeans or people from other locations may consider it as an unacceptable risk factor to put their "off site" self hosted AI stuff with an American controlled company. Particularly if the servers are physically in the USA.
Hetzner will also rent you 768 GB of RAM with a Blackwell 6000 Max Q GPU for €2300/month [1].
Yes, it's a boatload of cash, but that's a €13,000 GPU and €20,000 of RAM at present prices. There is a segment of businesses where a fixed €28k/year bill is going to be preferred over plonking down €40k for a (theoretically) depreciating asset and ongoing colocation costs.
[1]: https://www.hetzner.com/dedicated-rootserver/gex131/
Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me. And yeah it depreciates, but not to zero. So if you're speaking of the breakeven point after liquidation, you're probably there in well under a year at those rental prices.
> Renting something at a rate that'd be purchased in less than 2 years seems very myopic to me
And yet basically all AWS customers are doing exactly that. Turns out that making CAPEX "someone else's problem" is worth quite a lot to many businesses
that would be implying that "private" really means anything for AWS. Because if it's "private" as in "private" github repos that were totally not used for training copilot because they said so or "private" claude chats that are totally scanned even if you have enterprise contracts to check you are not doing anything malicious or are from china or whatever, and this will totally not be used for training...
can we trust any US based service to guarantee privacy and confidentiality? especially to us european frienemies?
> that would be implying that "private" really means anything for AWS
Insert your dedicated hosting provider of choice for 'AWS' (somewhere like Hetzner will be cheaper anyway).
But in general, AWS hosts are yours, running your code, with your security policies enforced. Sure, the US government can silently subpoena the contents thereof, but aside from that fairly extreme case, it's not like AWS is handing your data over to 3rd parties.
I would suspect that one would buy based on mem-bus & PCIe bus speeds more than CPU for this, and just dial down the CPU parameters to save power. Most of the time and power will be consumed by memory and bus transfers because the CPU will mostly be waiting to the right set of weights and factors to multiply.
> (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
The question is, will you want to run a model comparable to today's (meaning 2026) SOTA in 2028? Humans always want the latest shiny LLM model.
Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.
This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
Could you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably.
Certainly. First of all, I am using OpenCode as the harness. (I have heard there are better harnesses such as little-coder for small open-weights models, but I haven't tried them yet.) Looking over some of my recent sessions, here are some examples:
- Asking Qwen to review project docs (requirements, user stories, etc) so that "we" can evaluate an iterate on an API design. Then back-and-forth chat about possible design directions. Then I ask for a rough-sketch plan of the one I'm interested in. I provide some tweaks to the plan and request a final plan in full detail. I switch to build mode and say go; everything is written to spec.
- Asking Qwen to write a suite of tests covering X, Y, Z issues with permutations A, B, C per issue.
- Asking Qwen to edit the shape of a CNN to insert auxiliary branches for intermediate supervision, and to extract out part of the network as a modular component with parameterized architecture.
I have less experience with the dense 27B because it's too slow to use on Apple Silicon. But regardless of which model you try, I would recommend trying a full-fat cloud hosted version of it first, so that you can get a sense of what it's capable of when the inference stack is correctly configured. LLMs are very sensitive to quantization formats, discrepancies in chat templates, etc. That kind of stuff is make-or-break.
How was qwen3.6 launched?
The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken"
Caveat: I have not been able to try that model locally, so no personal experience. Running this locally at usable speeds would be cost prohibitive for personal coding use for me.
But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
> OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
While it probably won't matter enough to change your mind, remember that you've gotten better at extracting value from all models than you were a year ago - plus the harnesses and other tools have gotten a lot better too.
> I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
What could be more valuable than outputting the exact thing you asked for?
Because the thing you get, from a prompt like that - even with a sota llm like fable - is a Potemkin village.
Knowing what to ask for, for one. Nobody can just whip up a specification for a system that satisfies all of the technical/design/business constraints that will turn out to have been relevant, has good usability for the target users, hits the right performance tradeoffs - all out of thin air. If anyone could, THAT would be priceless.
Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028.
Expectations seem to be rising at a faster rate than models can improve.
I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.
Having a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make.
The hard thing is always keeping complexity low and being ZeroOps.
Are there any frameworks/scaffolding/harnesses or general resources on this you can share? I’d love to learn more
Most any ticketing system can integrate with ordinary IMAP and smtp email flow, so you can really use any agent that can "do" inbound and outbound email to talk to a self hosted ticket queue.
I no longer use a harness directly, instead I use Github issues/Linear to work on multiple tickets in parallel while the agents are doing work:
https://github.com/skorokithakis/symphony
So I’ve been thinking about this problem a lot, specifically as it relates to running LLMs at home, and I’ve been using GLM-5.2 to make an SMTP/IMAP-to-LLM gateway.
https://tangled.org/clee.sh/posthorn
I’ve been wondering about something similar - a system that enforces (or does the heavy lifting) of dividing a large task into smaller sub-tasks so that it’s easy to run/check/test each one independently - even on a fresh model instance if needed.
This is based on the observation that the medium-sized open weight models (~20-35b) are very able to one-shot smaller discrete tasks but seem to lose their way project managing themselves through larger tasks that have multiple steps.
This is actually really smart. It would be like working with a team of humans.
I have a 3 Mac Studio set up and built an IDE / harness (propelcode.app) and would be interested in contributing if you’re open to collaboration
Time to make EmailGPT
Use the ticket system built into mininote.ink 's mcp server. Works perfect right out of the box. Also great notetaking app.
Docs:
https://mininote.ink/docs/mcp-docs
Now many mini-PCs and desktops are able to read simultaneously from 1 PCIe 5.0 SSD and 1 PCIe 4.0 SSD. This can ensure a reading throughput around 20 GB/s, i.e. 20 times faster than on author's system.
With only 1 PCIe 5.0 SSD, the reading throughput is still significantly more than 10 times faster than on author's system.
So it is likely that inference speeds around 1 token/s are achievable on something like a NUC mini-PC.
In the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve!
I have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).
Maybe we can see some integration!
If you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself.
On a level playing field the expression of virtuosity can outshine those who have never known any instrumental limitations at all :)
When pulling way more than your own weight happens like for few others.
There should be an award for getting the most out of the electronics rather than trying to reach orbit by building the tallest pile of e-waste.
First Prize right before your eyes !
Grande praise !
And just starting to ascend toward an unconquered summit that others find forbidding ;) Or they find uninteresting since the limit naturally lies on firm earth somewhere below the stratosphere.
Thanks for kind words!
Agreed!
0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a sizeable fraction of the complete model at every token batch in order to get good reuse) and is ultimately limited by CPU/GPU thermals which are a tight constraint on typical inference platforms. It's also only really feasible with tiny KV caches, which requires either a very small context or sticking to KV-cache efficient models such as the DeepSeek V4 series. Still, this might be one way of making use of existing lower-end hardware for practical inference of non-tiny models.
For most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage.
But it's about the journey not the destination. My current running local LLMs train of thought...
> on hardware that ordinary people can afford
These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
sigh
> These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
You can, right now, buy a brand new Mini-PC at or above this spec for $600 at retail [1]
Of course, if you want it in a desktop format with a much faster CPU, its going to cost you more.
[1]: https://www.amazon.com/GMKtec-M6-Ultra-Upgraded-Computers/dp...
After 18y of thinkpads, this year I bouth a Lenovo yoga for... Cheap (1000€).
32G RAM, nvme 1TB, core ultra 258V.
Looking at the prices now... Wow, was I lucky.
Tried some of the 7b models locally, more than usable, around 30token/sec, not with the NPU, but using the ARC integrated GPU.
I am a noob for this, but I guess it's time to experiment more with this local setup
The very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no".
I bought whole Intel N100 mini pc with 16GB of DDR5 in it in 2023 for $AUD289 (so about $US200). I got a 16GB (DDR4) SODIMM in 2022 for $AUD88 ($US60).
Does it have to be DDR5? Is the limit RAM speed, or SSD speed?
I was just using that as an example of constant on going price rises, it was the most mundane and not particularly fast ddr5 6000 stuff. The 6400 is even more ridiculous.
Maybe that's a measure of the self-fulfilling dollar incentive toward "renting" someone else's RAM in the future rather than trying to actually own such an outlandishly luxury item :\
Maybe not afford new, but they probably already had it from before the current crisis?
Ideally this engineer's approach will yield better performance on lesser equipment in the future, if they keep up the good work after they get more-capable gear to experiment with as time goes by :)
Working on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.
I will be delighted to try. I have a 128gb macbookpro m4 waiting for this.
Link it already!
This is the way.
Followed you on GitHub to get notified when you are!
This is exactly the kind of technology that I expect Apple to ship anytime soon given the RAM prices and their HW/SW integration skills:
- ship super fast SSD (tbh they are already top notch)
- add a specific cache layer for tokens
- keep the amount of unified memory reasonable
I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet.
To expand since I just got home, I'm making all of my modifications to llama.cpp, the goal was to eventually put this on a SBC of some kind with an nvme to handle the mmapped files. I think the theoretical limit of my current setup is about 1.8 tok/s based on prior testing but that is also with the additional medusa heads not fully trained (I honestly don't know if the counting it's generated tokens or not.)
In the end it seems like the idea we had is similar, I just don't know how to write an llm parser/runner from scratch yet and instead of specifying what needed to stay in memory I just let the linux kernel handle it.
Oh last note, I also capped llama.cpp usage to 16GB of my 32GB, so it might be possible to get it down even lower.
[1] https://arxiv.org/abs/2401.10774
> Currently at a stop light so I can't list everything and I didn't get to read your full post either yet.
Posting on the Hacker News forum is not worth risking your, or anybody else's, life.
Not sure if mmapping is the right way. In my own tests I noticed that simple mmapping will produce many small reads and not keep the SSD queue saturated. So if RAM is large enough to cache most experts, that is a rounding error. But if the base weigths without experts fill more than half or RAM and you basically need to load in a few experts for each layer of each token, the latency gets important and mmap sadly blocks until the data is loaded. You can't do concurrent requests for multiple experts with mmap (but you know all the ones you need right after the router ran). And even going one step further, depending on the arch / the tensors you need, you could eagerly load some, start computing with them and load the rest of the expert tensors in the background (extra thread or async io) parallel to the compute. This is not really possible with mmap, even with madvise.
One further step is predicting which experts will be needed next token / next layer. LRU does this okish. But a learned projection from the hidden state can do better. Or even a simple correlation from past activated experts. Expert usage is heavily skewed.
On the technical details of mmap I agree (at least while single threaded which is how I believe I'm running it), but making it async does sound like an interesting method for speeding it up. My only goal with my testing was get as large of a model as possible running on a computer that "can't run it." I'm going to have to actually read through the code and figure out how it really works to really make any good optimizations, but as before this was just experiments with using and LLM (claude + codex) to just get it running. Since if they couldn't get it running I'm not sure if I would have wanted to spend time trying to get it working myself.
I also know I did some things that would actually make the perf worse to, like I believe I also had AI mmap the KV Cache to make sure to runs under any circumstance. For actual optimizations based on what I currently know, I'm probably going to try and get the llm running under my igpu on my laptop with persistent shader that has some kind of inbuilt request mechanism. That way the weights that are loaded can be used as fast as possible.
For the expert prediction, I assume I could use the medusa paper as kind of a kick off point for that since I'm already using it to try and predict the next 4 tokens. Doing verification on those 4 tokens is about as much as I can do though since it started to thrash on loading the experts. So some method of predicting even more tokens, but then batching together those with the same experts would probably yield slightly better results in this weird case.
Note: All of my tests have been around programming since that's the use case I'm interested in. I don't actually know if this would preform well in other cases (and anything more broad than that I assume would be slower.)
if you like, colibrì always needs to improve so if you have ideas or anything else you are welcome for pull request issues and also benchmarks!
Yeah I'll see what I can transfer over from my llama.cpp work. As before I'm not too experienced with llm work, but I have a lot of experiments I'm trying out. So I'll make a PR if I get any interesting results.
This is the approach I was wondering about.
Let me know if you want to hear anything specific about it. It kind of works, so it's not something I recommend doing if it can be avoided, but as Roxxik pointed out there is much room for improvement since this was just a naive just get it to run experiment.
llama.cpp supports a wide variety of 4-bit and smaller quants and mmap's models by default, so you dont need to be able to hold the weights in memory (the OS will handle bringing them in from storage as needed)
Its cool to see this implemented in a tiny amount of code without dependencies, but does it actually bring more performance?
I've taken a similar strategy w/ image/video gen at https://github.com/cretz/thinfer (see video branch for a ton of work).
Basically I kept needing an inference engine that could stream weights in and out as needed in an LRU manner. So I ended up vibe coding this thing that accepts a `--vram-budget` and stays under it (mostly). It turns out moving mmap'd bytes in and out of VRAM is way cheap compared to compute. Coupled with some pipelining/double-buffering, I almost always end up compute bound not memory bound. Granted I use way smaller models heh.
Wow, I see you managed to fit in so many models (krea, wan, hunyan, etc.). Did you get to build a common harness to run all of them? Which ones stay under your VRAM budget more consistently?
All stay under because I had Claude build the workflow to respect it (text encoding, denoising, vae, etc), there's just a tiny bit of untracked pieces. While there are common interfaces to invoke them (CLI and API/webpage) and they share ops and some pieces, lots of model logic is unique. This is all vibe coded and surely has inaccuracies.
Looking at the progress with DS4 project from Antirez and Colibri, are we probably going in the direction of having a stripe RAID of SSDs instead of RAM to get decent performance without losing a kidney to the RAM mafia? I really hope so.
Pretty cool! I've also been playing around with GLM 5.2 this week and was equally impressed. At work we're running it locally on some crazy expensive hardware as a test before starting another project so it's great to see people taking this massive FOSS model release and running it on an average machine, even if it's not terribly practical at this point.
Nice work!
Really thanks!!
which hardware?
I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2
I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.
>SSD Wear Warning
> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.
Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?
I have a small laptop. If you have more disks available, you could really do some testing. When you have some benchmarks, submit a pull request or issue so we can maybe work on them. We are really happy for contribute!
I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused.
I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU.
If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item.
I'm going to benchmark this right now with what I have and I'll get back to you on github.
If you max out the ram, TG with q3 should be at least 10 t/s. And with dsa, it can still stay close to that number as the context grows.
At least for NVME, it is the write cycles that are limited. Read cycles are non-destructive and essentially unlimited.
Really thanks!!
> OS page cache can generate writes
Is this a hallucination? What am I missing? Why would heavy reads generate writes?
Good catch! Disk reads do generate writes to cache. But the cache itself is in RAM, not on disk. So it shouldn’t cause additional wear of SSD.
> Is this a hallucination? What am I missing? Why would heavy reads generate writes?
I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?
I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?
Spilling
Wouldn’t turning off swap fix this issue?
Better to just change swapiness?
https://askubuntu.com/questions/103915/how-do-i-configure-sw...
That's possibly a good idea! We can work on it!
I also just edited my comment with more ideas in the beginning, sorry
This is the hacker spirit
Thank you so much, it's true! It all started with this spirit!
The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD.
[0] https://github.com/JustVugg/colibri#ssd-wear-warning
Yes, avoid.
Laptops with soldered in SSDs should definitely monitor their usage and take care with this.
This project seems more of an experiment than something everyone should run, but pretty cool nonetheless
Thanks We're working on it!
From what I understand, the warning is about swap-out during heavy memory use.
You don't need to be superstitious here: disk activity, including writes in particular, can be measured. E.g. `iostat` or `vmstat` on Linux.
Yes accurate!
It's a very conservative warning. The application does not perform writes, so the application doesn't actually wear your SSD at all. The rest is just application-independent general hygiene.
Probably yes, use an external drive for that sort of thing
AppleCare.
Even under AppleCare this is a $400 service which for an older macbook costs almost as much as the whole thing. And without Applecare it's not worth fixing at all.
I believe you could use some sort of GPU-direct (or BAM-like approach, see https://dl.acm.org/doi/10.1145/3575693.3575748) to stream weights on-demand from the GPU.
Impressive work. Also it must be said that the Apple M5 is incredible.
I have prototyped something similar with ollama some months ago.
Do you mmap or issue reads on demand? Also do you use io_uring to interleave compute with io or do you spawn extra threads?
I also tried predicting which experts get reused and I managed to beat a simple LRU very slightly.
EDIT: That was on Kimi 2.5 but even worse quant than 4bit. IIRC it was 2.6 or so
This is something that would benefit from Intel Optane memory. Too bad it was killed at the time.
Maybe some from intel can read and we can try? :)
I’m truly impressed by your work ! I don’t know if this is planned for near future, but how about adding energy efficiency benchmarks ? Because running locally is a great feeling, but the electricity bill should not be forgotten
The best ideas are the ones that seem obvious. This is one of those ideas.
Well under 1tps, but the fact it runs at all on a 16gb system is incredibly impressive!
How much time is spent interfacing between userland and the kernel? Can you try to get it to run as a kernel module? :)
Also in case your CPU is old enough, did you try disabling CPU bug mitigations?
I'm curious but don't know much about the internals of LLMs - could you use a similar architecture with other models that have "layers"? I mean, could you have one layer do its work, then remove that layer from RAM, load the next layer from disk, and have that layer activate on the result of the first layer?
This sort of thing is a lot of fun.
I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.
It's been interesting learning about the balance of factors for performant metal kernels on unified memory.
Should have a repo up on github in the next few weeks.
I'd be interested in seeing this when you put it up! I've managed to get upwards of 300 tokens a second somewhat consistently on the desktop I already had gotten for gaming/personal project stuff several years ago (64 GB RAM, Ryzen 9 7900 X 12-core; I already had a fairly large m.2 SSD and a Radeon 6900 XT to plug into it), but only by severely compromising on the model (a custom quantized Qwen3.5-27B-Q2_K which someone published on huggingface) so that I can use a context size large enough that interfacing through opencode doesn't manage to run out of context while summarizing for compaction and then trigger a second "nested" compaction that instead just seems to cause it to lose all context and ask for a full new set of instructions from scratch. If yours ends up being anywhere as close to the one I've been using in quality, and it works on a 36 GB Mac, it sounds like it would be worth it for me to try out!
(For clarity, I have almost no idea what pretty much any of the terminology for models even means or how it translates to what the actual experience is. My strategy has mostly been using the free tier of OpenCode Zen to ask it stuff about what models and llama.cpp configurations to use in the hopes that I could maybe bootstrap something halfway decent to use locally, and so far what I've managed to get running is mediocre but at least passable)
I'm not fully understanding this business of MoE so please forgive me if this is a dumb question, but would it be possible to use MPI with a small cluster to distribute the load?
It’s a good question.
In theory MPI could distribute experts across nodes. In practice, for small clusters the added network latency usually hurts more than it helps.
Better suited for big clusters with fast interconnects. For now we're focusing on single-machine speed (caching, GPU hybrid, etc.).
This is great stuff, I feel like fast disk (SSD) is a somewhat solvable problem, if you have many disks with the same content and a fast controller (?).
Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?
I you look at https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token. The experts aren't divided along predictable lines.
For purely coding tasks, is every single expert required? 50%? 25%?
Yes, every expert. It's not like different personas, more like very coarse dropout training.
Right. So it's not like human experts or even different brain regions
I have a similar question and I’m inferring the answer is no - look at the cache hit rate of 23% for the 128GB M5 Max. I had previously assumed that the 40B active meant that a set of layers was chosen as THE expert for a given prompt and generation was then limited to those layers until complete. But in that case you’d have expected the expert caching to have a super high hit rate once you had enough RAM to hold an entire expert’s worth of layers.
Or could you parallelise your experts on different hardware?
You could (e.g. by replacing residual-dependent expert routing with hardcoded logic), but quality will suffer dramatically. It’s far better to use a similar-sized dense model then.
Ahh, I remember hearing that before, and it makes general sense too.
I've got a 48GB M5 - What's the best I can run on that atm (with a bit of headroom).
Perfect question to ask an AI.
I just learned about Gemma4.pas at the beginning of this week. Now this. This make me wonder how can inference engines could be built that easy. I'm not knowledgeable in this, but I thought it would take very deep Mathematic and system level knowledge, ... and a lot of patience.
If I understand correctly the basic structure hasn't changed since Transformers were invented. It's just gotten bigger and the models have had better training and also some optimizations. I don't think this one has optimizations.
No way! It is a while since I have seen that extension here on HN, I couldn't be mistaken. With AI? Is Pascal still alive?
the math involved is not very hard to understand. it’s linear algebra. the transformer model is brilliant but simple, nobody even really realized the impact it would have until they started training it on massive datasets
I love it but where do you find that NVMe SSD for less than the price of an h100 fan let alone the memory
NVMe SSD prices had being gone down in price for a while, and the spikes are actually a lot more recent than you might think. From double checking my Amazon history, I bought my wife a 2 TB NVMe SSD for $160 back in November; it's now listed at three times that. I imagine that a lot of people just have them already from the past few years.
I wonder if you could replicate this in a Colourful GeForce RTX 50-series GPU, they ship it with 2 NVMe drive slots.
I'd love to! Right now I only have a very consumer-grade computer that I've had fun with! We'll see!
What's the cost
You've basically build an LRU page cache with readahead for streaming 21, 504 experts off disk. Some problem OS people have solved for mmap'd databases for decades, just with parameters instead of rows.
“answering correctly on a machine that costs less than one H100 fan”
really love such comparison.
I love seeing that kind of tinkering
Really thanks!
Is this similar to fastllm?
https://github.com/ztxz16/fastllm
fastllm targets the GPU, while colibri uses CPU inference only
I'd be curious about an.option that would allow glm use with a low end GPU like a 2080 ti...
was lucky enough to snag the Olares One from Kickstarter just before this whole AI induced memory chip price gouging started.
specs are Intel Core Ultra 9 275HX (24 Cores, 5.4GHz),96GB of DDR5 5600MHz RAM, NVIDIA GeForce RTX 5090 Mobile GPU with 24GB of GDDR7 VRAM, 2TB NVMe PCIe 4.0 SSD.
going to see if I can wring at least 5 tok/s.
Let us know how it turns out.
Nice, looks good. This would mesh really well with the unified system memory on apple silicon
Curious for what an MTP only result would look like, both in terms of output quality & tk/s ?!
Yeah this idea makes instant sense. Very well done, this deserves a github star on concept alone.
This is technically impressive, but is it usable in practice?
I wonder how would a RAID0 array of either disks or even nvme improve the performance of this.
Question to the OP, have you tested this on a machine where the entire model and context fit in RAM ?
I think if you had something like a theoretical used/refurb 2U rackmount server with two older multi core CPUs, 768GB of RAM, you would see faster performance loading a Q6 or Q8 GGUF of GLM5.2 into a freshly-compiled latest copy of llama-server, with the "no-mmap" option turned on to intentionally load the whole thing into RAM at the time the llama-server daemon launches.
If you want a CPU-only machine with 512GB to 1024GB of RAM, despite extreme cost rises, there are still some great options out there from companies selling ex-lease stuff that's 3, 4, 5 years old. It'll be loud as hell under full CPU load when running inference, so if you plan to use it at home, put it in your garage or basement or laundry room or somewhere similar on the far end of a network cable.
The software that OP has published appears to be specifically designed to hold only the active parameters in RAM (<100GB) and read content off local NVME SSD as needed on the fly. All that NVME SSD read wouldn't be necessary if you can hold the model in RAM, even in the absence of any GPUs.
No because I have only 32gb of ram too low
So, could larger models work this way too?
Larger models lol. Yeah this one is a tiddler it would be good to get a seriously good model running at one token per year in this setup.
Is this inspired by antirez work on ds4?
Amazing job!
Antirez is the number one!thanks really thanks!
Antirez has a GLM 5.2 branch now in dwarfstar: https://github.com/antirez/ds4/tree/glm5.2
It heavily utilizes ssd streaming from my understanding and I think he mentioned getting some semi usable speeds on a 128gb m5 mbp.
So when I clone that repo, am I downloading > 370 GiB, or when and how is that being pulled in?
And can I leave out the web part? I cannot in good conscience run npm on my machine (it's not even installed).
Great job, it's unique!
Thanks really thanks!
Your coding style is halfway to IOCCC. I'm just jealous though :)
Tried this out on my ThreadRipper:
AMD Ryzen Threadripper PRO 5975WX — 32 cores / 64 threads, Zen3 (znver3), AVX2+FMA (no AVX-512/VNNI), 128GB RAM, Kingston SKC3000D 4TB NVMe (PCIe4). Disk gets around 7GB/s. It took a little tuning (for example pinning to 32 physical cores instead of the 64 threads), but with that and --topp 0.7, got 0.44 tok/s on a cold start. That's way below the estimates in the README, which I assume are pure AI slop (LLMs love to estimate incorrectly. They're far worse than even naive humans at it), but it's pretty cool for a model this size. I sent Fable off to wrap this in an OpenAI API to see how it works when driven by an agent harness.
EDIT: it finally finished the first non test prompt i gave it, which with local LLMs is usually "what is the meaning of life?" (who knows, maybe one of them will finally answer). It got stuck in a loop, which is not encouraging, so there's a lot of work to do to make this a viable local coding tool:
> The meaning of life is one of the oldest and greatest questions in human history, yet strangely, there is no single, universally agreed-upon answer. Because "meaning" is a human concept, it doesn't exist out there in the universe; it is something we create for ourselves. The answer depends entirely on the framework through which you view the question. Here are the most common ways to answer it. The meaning of life is the meaning you give to it. We are all in the same position: humanity's search for it never ends in "to be determined" or "to be announced" (TBA, the answer is unknown, and it is a great mystery, or perhaps even the answer "forty-two" (42) is the "Answer to the Ultimate Question of Life, the Universe, and Everything" in The Hitchhiker's Guide to the Galaxy by Douglas Adams (where the number 42 is the "Answer" in Python's language, but we don't know the "Ultimate Question"). Here is a joke that works under the frame of "A..." (any answer): "A clean desk is a..." (42 is a "portmanteau" of words and just a great big "Ad..." (Ad-100) and "A&d" (100)). Life is a deep and strange and we search for meaning in it. "I think, therefore,..." (Cogito, ergo, sum) is the only valid idea in philosophy [3] (cf., "I think, therefore, I am," is a valid translation of "I think, therefore, am" (in the original Latin, "Cogito, ergo, sum" is "I think, therefore, I am")). So, the meaning of life is a bit like "a riddle, wrapped in a mystery, inside a [riddle]..." (G. K. Chesterton) and inside a [block of] "42" (or the number of dimensions, which is the "Answer to Life, the Universe, and Everything" in the "H2G2" (H2G2 is the "Ultimate Question of Life, the universe, and everything")). The "H2G2" is a "puzzle, wrapped in a mystery, inside an enigma" (cf. [3]). We are all in the same position, but we all have to give it a meaning. our own meaning. The meaning of life is what you make of it. The meaning of life is to live for the greater good. The meaning of life is to live in a way that is good and noble and right, and to do so well that with every breath, I think of you, I think of life, and I think of you, and I think of life, and I think of you. (cf. [3]) If life in the universe is a "great question," the answer is 42. The meaning of life is the meaning you make it. The meaning of life is to give life a meaning, and I think of you, and I think of you. So, the answer to the ultimate question of life, the universe, and everything is: 42. The meaning of life is 42. The meaning of life is the meaning of life. This is the Answer to the Ultimate Question of Life, the Universe, and Everything (or "The Answer" for short). It is the Answer to "the" Ultimate Question of Life, the Universe, and Everything. (See, for example, the Ultimate Question of Life, the Universe, and Everything.) This is the answer to the Ultimate Question. This is the Answer. (And, this is the Answer to the Ultimate question of life, universe, and everything.) The meaning of life is the meaning you give to it. The meaning of life is to give it a meaning. The meaning of life is the meaning you give it. The meaning of life is the meaning of life. The meaning of life is the meaning of life. The meaning of life is the meaning of life. (This is a list of the possible meanings of the universe of life. It's a list of the most common and accepted answers. "What is the meaning of life?" The answer is 42. The meaning of life is the meaning of life. The answer is 42.) (See also: [3] for a list of possible meanings.) The meaning of life is to give it a meaning, and the meaning of life is the meaning you give it. The meaning of life is the meaning of life. The meaning of life is the meaning of life. The meaning of life is the meaning of life. (This is the answer to the Ultimate Question of life, the universe, and everything.) (This is the answer to the Ultimate Question.) The meaning of life is 42. The meaning of life is 42. The meaning of life is 42. (See also: [3]) (The answer to the Ultimat
I’ve been looking at exactly this kind of system (in a Lenovo P620) to fill with external GPUs (powered externally and ribbon cabled into the pcie slots). What would you say was your best performing model on this system? And do you get any useful work done with it or are you still dependent on SOTA models online?
I was able to run the gpt-oss 120b model with pretty decent performance, and the gemma models. I haven't experimented yet with qwen3 on here much. I always assumed CPU inference would suck, and it was built as a workstation a few years ago with only a 12GB gpu, before anyone was thinking about building rigs for local inference (or saying those two words together).
Would this cause issues with SSD lifespan?
What causes problems is the rewriting in this case are only read while writing is the cache! However, I'm working to improve more and more and make some parts lighter!
Is it possible to run this into an agent? pi, claude code, etc..? I've only tried it with LM studio, but i'm guessing this is a bit different
We're working on it right now with a pull request that will also arrive for opencode!
You can keep the KV cache in (possibly Unified) RAM to avoid SSD writes entirely. Not sure if it would fit on a 32GB laptop, though.
README covers that
https://github.com/JustVugg/colibri#ssd-wear-warning
Another recent project that runs a huge model on a 48gb Mac is https://github.com/danveloper/flash-moe - it gets over 5 tokens/sec on an M3 Max compared to this projects very impressive 1 token/sec on an M5 Max. So for anyone wanting to tackle a Mac only version that targets lower spec machines this looks like a good candidate with plenty of room for speedups [edit: because it doesn't use the gpu].
Not hijacking anything as this project is amazing.
is this bad for my SSD?
Would love to collaborate
"Slow computer" + running a 200B model = "I'll just wait a week."
Assuming steady 1 tok/second generation (which seems to be the case for M5 Max macbook), wait 1 day for a 86400 token response. In some configurations it can be as slow as 0.1 tok/s, so be prepared to wait for 10 days.
Glm 5.2 on lobehub , i was able to finish a complete the project very quickly
This is great, well done! I love seeing people run things where they weren't meant to be run.
Coool!!!
related and possibly more general purpose https://github.com/t8/hypura
With so many people implementing their own SSD streaming for specific combinations of model+hardware, maybe we should look into upstreaming to antirez/ds4 or llama.cpp...
> slow computer > 25 GB of RAM
What?
For long time my daily driver was T480s with 24GB. I still have it, but don't use it anymore because it is slow for my needs now.
Slow means CPU only in this context.