The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context.
At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.
You need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps.
I don't see LLMs being production-ready until the context rot and sampling problem is fixed forever. This has not occurred, and the big inference providers aren't even bothering to integrate any of the research on that subject.
If anything, many of the bigger companies are actively making inference quality worse just to extend their runway a tiny bit farther before they go bankrupt.
The only thing the article gets right is this: if you're serious about LLMs, abandon Big AI and infer locally only. This is the only way you have control over the quality of the output.
Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out.
Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance.
How do you all keep your context so low? Complex tasks just balloon it in my experience.
The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases.
To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too).
To make your prompts more precise, tell the LLM which files it has to read (or at least where it should start), so it does not have to search as much. This also reduces the change of misunderstandings and makes the LLM adhere to existing practices.
To keep your code base in check, tell the LLM (in a new session) to review the code and refactor from time to time.
When a task is done, start a new session. If you find that you have to repeat a lot of information in your next prompt, put the information in a file so you can reference it in the future (aka documentation).
I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models).
I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has.
Lately I haven't been saying go to X file and change Y, I say you have SSH keys to the embedded hardware: plan, deploy, and test latency improvements to the X stage of the image processing pipeline, parity test output images to avoid regressions, and target a 5ms latency improvement, ideas to look at are M, N, and P image processing techniques, I think latency is in this domain is mostly caused by Y, but verify, and let me know when you have a plan ready to implement.
Then it churns for 4 hours, it builds and runs it's own test harness and profiler, and autonomously works until it has a changeset ready to go.
That's how I get to 600k, and it seems to work fine? It got us 4.8ms today, through SIMD improvements and using a technique I hadn't thought of in place of a more expensive operation.
Sure it works fine because you havent yet experienced how much better it can be. I encourage you to stop after the 'plan' part, read the plan, and tweak it. Have part 1 of the plan be it coming up with a framework to evaluate success, part 2 be it spitballing solutions, and part 3 be taking the best solution and "productionizing" it. You will get much tighter output and the brainstorming wont all muddy each others context as it tries different solutions. Bonus points for having part 2 fan out on its own, though thats much harder on shared hardware.
> Sure it works fine because you havent yet experienced how much better it can be.
Better in what way, though? It solved my problem, without issue, in a very elegant way. Perhaps I could tweak and optimise to make it solve it slightly faster but honestly, I'm just not that into it. It works, idk, doesn't seem like it needs fixing to me.
But honestly, I don't care a lot about these tools, I said elsewhere I'm probably switching careers if this is the new norm, I hate this version of engineering. It's not even engineering, it's just management.
Only going all in because our company is forcing us to.
I stopped using GitHub Copilot extension in VS Code when they introduced their new pricing model, but I should have switched much earlier. The developers have barely a clue how LLMs work and the company structure is misaligned with creating a quality product. They do not perform benchmarks to evaluate whether new "features" are any good and instead bloat the context with more and more tools that are rarely useful and often confuse models.
Eventually, the context window got so bloated that they resorted to hiding function bodies in large files, which is of course a stupid idea because then the LLMs have to use other tools to read the files, wasting even more tokens, or hallucinate the content. Honestly, it is amazing that LLMs work at all in VS Code.
While this can work, especially for disposable code, doing this for non-disposable code is going to set you up for a very very _very_ rough awakening, the moment the models fail or your access to them fails.
Make sure to regularly validate that what you think you understand is still what you actually understand.
One can only safely offload so much.
I mean, thanks, but the PR was 60 lines of code and I've been doing this for over a decade, so I know how to critically evaluate the output.
I don't want to do this, for what it's worth. But my company has gone all in, I'm pretty sure our C suite has psychosis, and if I don't use it I'm seen as lesser.
I'm not planning to do this much longer. If I can't find a real SW engineering gig again soon I'm switching careers (to what - idk yet). Capable models or not, I don't want to be a manager, I want to solve problems, and they took the best part of my job from me.
I spent the time it was churning practicing drawing.
Pretty much my experience too. Using Sol in VSCode I just give it the problem and let it go. I don’t even look at or think about context. Works well for me.
> your prompt is confusing, unfocused, and doesn't work right on any LLM
You are assuming the entirety of the prompt is human prose, but it could be sets of data so the agent doesn't have to collect it every time, like program interfaces, commands, views, databases, tables, data models etc...
I could see this scale to multiple kiltobytes of metadata in the prompt easily.
That doesn't make much sense to me because this is in nature much like how harnesses operate: launch a bunch of exploratory subagents to search and retrieve evidence to use in the actual prompt. Think of it as caching this end result so you don't have to re-fetch in the codebase.
That's the other way of doing it, which solves the context rot problem in a more complex way. The model at the top says, "hey, sub-agent, go figure out the answer to this question and give me the answer", and that sub-agent can go consume 250k+ context to return an answer that might be a couple of words, and thus not contaminate the main context with that now thrown-away context.
However, this is not something that is inherently part of models or inference engine, but part of the harness.
Harnesses are very hit and miss, and are not integrated into the stack, and I think that will have to happen eventually. Like, conceptually similar to an LLM performing a tool call that just calls itself recursively, I think this would go a long way to making LLMs more viable for being an actual product people could conceivably want.
Yeah, strong evidence for what parent says is correct. Been my experience as well, especially with local (smaller) models but also SOTA. The less instructions you have, the better they get at following them. Conflicting instructions is like poison, and it's harder to find those conflicting parts the longer the prompt is too.
the longer the context grows, the greater the probability that it generates ambiguity, and the probability that it makes mistakes approaches 1.
A system prompt should be looked at like a starting point and a direction, but not a giant atlas map of everything it may be asked to do.
I've been playing around with Qwen3.8-Next-Flash that has great logic, recall, roleplaying, etc. From what I can tell, it's definitely on part with the SOTA a few months ago, and my opinion, it likely is around the pinnacle of advancement without high inefficiency in preparing with the current LLM recipes. Further, I'm of the opinion America's SOTA is hitting the real cost-sigmoid and there's no singularity in site. These things will hack the planet if you put them in a group of agents and tell them to do it; but as context grows, the probability that they can answer "how many r's are in strawberry" goes down. No amount of parameters it going to erradicate that.
But I digress, my new stage of working with LLMs is figuring out how to use Qwen3.6-35B-A3B as the entry point to collect the context, and then use the big boys Qwen3.8 to make the edits, then degrade back and forth to complete changes. There's no harness yet for this, but there's clearly an intelligent way to setup a engineering harness.
And I do understand people have codebases that simply can't easily live in the smaller (~128k-256k) context windows, but instead of porting your codebase to another novel language, breaking it into context-aware components would make it closer to what these things can do.
And I'll repeat: I don't think we're approaching the singularity of self-recursion primarily because the LLMs will duplicate errors, context poison, and whatever else they encounter and there's no human who can sit around correcting it constantly. The American AI apparatus should cut their models like the Chinese down and work on real problems and stop imbibing the singularity-watts-are-all-we-need drugs.
3.8 Next Flash seems like it is the first model usable unattended coding on a single desktop PC. The context window issue I think is a non-issue, at least to the extent that I already run Opus and Fable with auto-compact at 200k tokens. Even though theoretically those models support longer context I've never seen them perform as well at long lengths and the cached read cost starts to get really out of hand.
Thanks, I hadn't seen OpenViking before. I do everything in Claude Code right now due to the cost advantage but between the advances in local models (especially Qwen family) and models like Gemini 3.8 Flash it may be worth revisiting.
> "until the context rot and sampling problem is fixed forever"
I agree, prompt adherence seems to get worse when operating on large inputs.
Does anyone have some notion of the SOTA with this? Can we expect big improvements by this time next year? (hopefully in open weights)
A lot of this is managed by the inference engine, and has nothing to do with the model.
Models that use, for example, sparse attention mechanisms are just trying to make the bad situation slightly less bad, such as using less RAM for context (thus requiring less context quantization) or using less bandwidth (thus running faster).
If people keep using temp, top-k, top-p, and min-p, and nothing else for samplers, we're ignoring ~3 years of sampling research that virtually eliminates the worst of context rot issues.
> Everyone who begins learning exploitation hits a phase of exploitability grief about 3 month into dedicated, practiced study. They hack something they didn’t think they had the skill to break into and it terrifies them. They’re smart enough to know that, relatively speaking, they are an idiot, and if an idiot can do this then nothing is safe. That feeling is correct.
I've been using Claude Code Extension in VSCode (no phone-home configured), backed by DwarfStar on a LAN local MBPro 128GB M5. The context bloat is horrendous, leading to 5-10 minute prefills.
I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do?
(I kind of hate to abandon Claude Code, as it seems to be the most capable coding assistant of the limited set of tools I've tried. But that horrendous context bloat is really painful!)
TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally.
I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.
Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post.
Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.
Understood, and I realized you'd posted this to HN yourself, so I felt a bit bad making the comment. I think maybe for me, this might have worked better if the motivation had been one separate post, and the details of the gotchas as a post of its own.
But I also think, if you're going to be limited to 65k token windows, you're going to have a really difficult time. Even 250k windows were cramped for me when that's all we had on Anthropic models. I just don't think a 65k window is going to be big enough for proper cyberdefence work, even if I totally agree with going local wherever you can. It feels like if you're defending against swarms of 1-10M context windows, you need to get as close as you can to similar. I've had to reach for Chinese 1m models instead because the American models just refuse me here in Australia.
Well, if you are serious about it and you have Strix Halo, there are better ways of getting more context and capability and speed. Lookup halogen for Strix
The most cost-effective local option right now, I think, is dual R9700. You can run a 27B dense Qwen at FP8 around with a full context and 2-3 concurrent sessions of 260K context. If you go down to an MXFP4, you get 4 to 5 concurrent sessions. And speed is on par with anything you'll get from hosted providers. You're getting between 60-80 for FP8 and 150+ tokens per second speed for MXFP4 and pp is 4K+. Lookup vllm radiance
There is also a lot of progress in running Qwen 3.8 next flash with dual R9700.
Obviously one gets less context and speed is a little bit less, but it's still very acceptable. Better than what you're getting with Llama on Strix Halo, that's for sure.
These are small dense models,meaning Qwen, have gotten capable and fast. And there's been a lot of progress in the area. So sticking with Llama you are not taking advantage of the hardware you have. And yeah, for Strix Halo, you should just look into halogen and you shouldn't be just using 96 gigs for the VRAM. You should give it most of the VRAM to the inference and connect to it from your laptop or something. People ar egetting 1000+ pp with qwen 3.8 Next Flash
Your hardware can do way more than 64k tokens context window, can't it?
And with Ollama it's very easy, superficially you just drag the slider.
I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now.
So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.
I think this is worth revisiting once we get some solid 3rd party numbers from the new mac studio ultras, which admittedly are a bit over $10k with a 2tb ssd + 256gb ram. I think I'd be seriously considering it if I had a $200/month subscription of some kind.
This situation has improved quite a bit recently, Qwen Flash Next will run on a $4000 PC and can reliably implement small features on its own (feels comparable to Opus 4.5). It's a bit slow but pretty effective.
The article doesn't really describe the problem: if your prompt is 35kb, your prompt is confusing, unfocused, and doesn't work right on any LLM, and is needlessly bloating your context.
At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.
You need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps.
I don't see LLMs being production-ready until the context rot and sampling problem is fixed forever. This has not occurred, and the big inference providers aren't even bothering to integrate any of the research on that subject.
If anything, many of the bigger companies are actively making inference quality worse just to extend their runway a tiny bit farther before they go bankrupt.
The only thing the article gets right is this: if you're serious about LLMs, abandon Big AI and infer locally only. This is the only way you have control over the quality of the output.
Tbf, didn't read the article because it isn't applicable to me. I don't use system prompts or memory, I just use models stock and write the problem out.
Is it really 250k? I had a long running autonomous Astra session today that got to about 600k and it finished fine with everything I asked it to do solved nicely. Opus 5 last week got to around 700k before I compacted between prompts, but also gave good performance.
How do you all keep your context so low? Complex tasks just balloon it in my experience.
The question isn't how you keep your context small, but how did your context get so big? A few common sources of bloat are long system prompts, unnecessary tools, unclear prompts, and scrawling code bases.
To reduce system prompt and tool bloat, use a minimal harness (I wrote my own, but I've read that pi.dev is okay, too).
To make your prompts more precise, tell the LLM which files it has to read (or at least where it should start), so it does not have to search as much. This also reduces the change of misunderstandings and makes the LLM adhere to existing practices.
To keep your code base in check, tell the LLM (in a new session) to review the code and refactor from time to time.
When a task is done, start a new session. If you find that you have to repeat a lot of information in your next prompt, put the information in a file so you can reference it in the future (aka documentation).
I think I'm just slightly higher level, it seems. I don't give it small tasks, I give it ticket level prompts and let it decide what to do. (It being frontier models).
I don't have a harness or prompt. Just VS code integration through my company. All my prompts are from scratch and the only context it has.
Lately I haven't been saying go to X file and change Y, I say you have SSH keys to the embedded hardware: plan, deploy, and test latency improvements to the X stage of the image processing pipeline, parity test output images to avoid regressions, and target a 5ms latency improvement, ideas to look at are M, N, and P image processing techniques, I think latency is in this domain is mostly caused by Y, but verify, and let me know when you have a plan ready to implement.
Then it churns for 4 hours, it builds and runs it's own test harness and profiler, and autonomously works until it has a changeset ready to go.
That's how I get to 600k, and it seems to work fine? It got us 4.8ms today, through SIMD improvements and using a technique I hadn't thought of in place of a more expensive operation.
Sure it works fine because you havent yet experienced how much better it can be. I encourage you to stop after the 'plan' part, read the plan, and tweak it. Have part 1 of the plan be it coming up with a framework to evaluate success, part 2 be it spitballing solutions, and part 3 be taking the best solution and "productionizing" it. You will get much tighter output and the brainstorming wont all muddy each others context as it tries different solutions. Bonus points for having part 2 fan out on its own, though thats much harder on shared hardware.
> Sure it works fine because you havent yet experienced how much better it can be.
Better in what way, though? It solved my problem, without issue, in a very elegant way. Perhaps I could tweak and optimise to make it solve it slightly faster but honestly, I'm just not that into it. It works, idk, doesn't seem like it needs fixing to me.
But honestly, I don't care a lot about these tools, I said elsewhere I'm probably switching careers if this is the new norm, I hate this version of engineering. It's not even engineering, it's just management.
Only going all in because our company is forcing us to.
I stopped using GitHub Copilot extension in VS Code when they introduced their new pricing model, but I should have switched much earlier. The developers have barely a clue how LLMs work and the company structure is misaligned with creating a quality product. They do not perform benchmarks to evaluate whether new "features" are any good and instead bloat the context with more and more tools that are rarely useful and often confuse models.
Eventually, the context window got so bloated that they resorted to hiding function bodies in large files, which is of course a stupid idea because then the LLMs have to use other tools to read the files, wasting even more tokens, or hallucinate the content. Honestly, it is amazing that LLMs work at all in VS Code.
You can inspect the context by pressing F1 and then selecting "Developer: Show Chat Debug View" in VS Code (https://github.com/microsoft/vscode/wiki/Copilot-Issues) and marvel at all the garbage that is in there.
While this can work, especially for disposable code, doing this for non-disposable code is going to set you up for a very very _very_ rough awakening, the moment the models fail or your access to them fails.
Make sure to regularly validate that what you think you understand is still what you actually understand. One can only safely offload so much.
I mean, thanks, but the PR was 60 lines of code and I've been doing this for over a decade, so I know how to critically evaluate the output.
I don't want to do this, for what it's worth. But my company has gone all in, I'm pretty sure our C suite has psychosis, and if I don't use it I'm seen as lesser.
I'm not planning to do this much longer. If I can't find a real SW engineering gig again soon I'm switching careers (to what - idk yet). Capable models or not, I don't want to be a manager, I want to solve problems, and they took the best part of my job from me.
I spent the time it was churning practicing drawing.
Pretty much my experience too. Using Sol in VSCode I just give it the problem and let it go. I don’t even look at or think about context. Works well for me.
> How do you all keep your context so low? Complex tasks just balloon it in my experience.
By breaking the problem into discrete steps and aggressively restarting the prompt from the current state after completion of said steps
AKA "divide and conquer", how we programmers been fighting ever bigger and more complex problems since probably forever.
My guess is your sessions are interactive conversations rather than planning a course of action then, separately, executing each piece.
> your prompt is confusing, unfocused, and doesn't work right on any LLM
You are assuming the entirety of the prompt is human prose, but it could be sets of data so the agent doesn't have to collect it every time, like program interfaces, commands, views, databases, tables, data models etc...
I could see this scale to multiple kiltobytes of metadata in the prompt easily.
That usually ends up being a poor use of LLMs, and is an unsolved problem with LLMs.
RAG was supposed to be the way out on that, and ended up being mostly abandoned.
That doesn't make much sense to me because this is in nature much like how harnesses operate: launch a bunch of exploratory subagents to search and retrieve evidence to use in the actual prompt. Think of it as caching this end result so you don't have to re-fetch in the codebase.
That's the other way of doing it, which solves the context rot problem in a more complex way. The model at the top says, "hey, sub-agent, go figure out the answer to this question and give me the answer", and that sub-agent can go consume 250k+ context to return an answer that might be a couple of words, and thus not contaminate the main context with that now thrown-away context.
However, this is not something that is inherently part of models or inference engine, but part of the harness.
Harnesses are very hit and miss, and are not integrated into the stack, and I think that will have to happen eventually. Like, conceptually similar to an LLM performing a tool call that just calls itself recursively, I think this would go a long way to making LLMs more viable for being an actual product people could conceivably want.
The Claude Code system prompt was >50KB, though I think they trimmed it down heavily recently. (The newer models don't need as much hand-holding.)
Yeah, strong evidence for what parent says is correct. Been my experience as well, especially with local (smaller) models but also SOTA. The less instructions you have, the better they get at following them. Conflicting instructions is like poison, and it's harder to find those conflicting parts the longer the prompt is too.
the longer the context grows, the greater the probability that it generates ambiguity, and the probability that it makes mistakes approaches 1.
A system prompt should be looked at like a starting point and a direction, but not a giant atlas map of everything it may be asked to do.
I've been playing around with Qwen3.8-Next-Flash that has great logic, recall, roleplaying, etc. From what I can tell, it's definitely on part with the SOTA a few months ago, and my opinion, it likely is around the pinnacle of advancement without high inefficiency in preparing with the current LLM recipes. Further, I'm of the opinion America's SOTA is hitting the real cost-sigmoid and there's no singularity in site. These things will hack the planet if you put them in a group of agents and tell them to do it; but as context grows, the probability that they can answer "how many r's are in strawberry" goes down. No amount of parameters it going to erradicate that.
But I digress, my new stage of working with LLMs is figuring out how to use Qwen3.6-35B-A3B as the entry point to collect the context, and then use the big boys Qwen3.8 to make the edits, then degrade back and forth to complete changes. There's no harness yet for this, but there's clearly an intelligent way to setup a engineering harness.
And I do understand people have codebases that simply can't easily live in the smaller (~128k-256k) context windows, but instead of porting your codebase to another novel language, breaking it into context-aware components would make it closer to what these things can do.
And I'll repeat: I don't think we're approaching the singularity of self-recursion primarily because the LLMs will duplicate errors, context poison, and whatever else they encounter and there's no human who can sit around correcting it constantly. The American AI apparatus should cut their models like the Chinese down and work on real problems and stop imbibing the singularity-watts-are-all-we-need drugs.
3.8 Next Flash seems like it is the first model usable unattended coding on a single desktop PC. The context window issue I think is a non-issue, at least to the extent that I already run Opus and Fable with auto-compact at 200k tokens. Even though theoretically those models support longer context I've never seen them perform as well at long lengths and the cached read cost starts to get really out of hand.
this is what I use: https://github.com/Opencode-DCP/opencode-dynamic-context-pru... ; this and openviking appear to be all I need to get a proper 'do anything' setup.
Thanks, I hadn't seen OpenViking before. I do everything in Claude Code right now due to the cost advantage but between the advances in local models (especially Qwen family) and models like Gemini 3.8 Flash it may be worth revisiting.
> "until the context rot and sampling problem is fixed forever"
I agree, prompt adherence seems to get worse when operating on large inputs. Does anyone have some notion of the SOTA with this? Can we expect big improvements by this time next year? (hopefully in open weights)
A lot of this is managed by the inference engine, and has nothing to do with the model.
Models that use, for example, sparse attention mechanisms are just trying to make the bad situation slightly less bad, such as using less RAM for context (thus requiring less context quantization) or using less bandwidth (thus running faster).
If people keep using temp, top-k, top-p, and min-p, and nothing else for samplers, we're ignoring ~3 years of sampling research that virtually eliminates the worst of context rot issues.
Attention is all you need.
And there's only so much attention to go around.
Friends Don't Let Friends Use Ollama https://news.ycombinator.com/item?id=47788385
and tell your friends about https://llama.app/ by Nvidia
> Everyone who begins learning exploitation hits a phase of exploitability grief about 3 month into dedicated, practiced study. They hack something they didn’t think they had the skill to break into and it terrifies them. They’re smart enough to know that, relatively speaking, they are an idiot, and if an idiot can do this then nothing is safe. That feeling is correct.
I've been using Claude Code Extension in VSCode (no phone-home configured), backed by DwarfStar on a LAN local MBPro 128GB M5. The context bloat is horrendous, leading to 5-10 minute prefills.
I've recently been exploring tools like headroom to help manage context, with some limited "success" (for some definition of success). What do others with similar setups do?
(I kind of hate to abandon Claude Code, as it seems to be the most capable coding assistant of the limited set of tools I've tried. But that horrendous context bloat is really painful!)
attention is the key
TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally.
I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.
Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post.
Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.
Understood, and I realized you'd posted this to HN yourself, so I felt a bit bad making the comment. I think maybe for me, this might have worked better if the motivation had been one separate post, and the details of the gotchas as a post of its own.
But I also think, if you're going to be limited to 65k token windows, you're going to have a really difficult time. Even 250k windows were cramped for me when that's all we had on Anthropic models. I just don't think a 65k window is going to be big enough for proper cyberdefence work, even if I totally agree with going local wherever you can. It feels like if you're defending against swarms of 1-10M context windows, you need to get as close as you can to similar. I've had to reach for Chinese 1m models instead because the American models just refuse me here in Australia.
Well, if you are serious about it and you have Strix Halo, there are better ways of getting more context and capability and speed. Lookup halogen for Strix The most cost-effective local option right now, I think, is dual R9700. You can run a 27B dense Qwen at FP8 around with a full context and 2-3 concurrent sessions of 260K context. If you go down to an MXFP4, you get 4 to 5 concurrent sessions. And speed is on par with anything you'll get from hosted providers. You're getting between 60-80 for FP8 and 150+ tokens per second speed for MXFP4 and pp is 4K+. Lookup vllm radiance There is also a lot of progress in running Qwen 3.8 next flash with dual R9700. Obviously one gets less context and speed is a little bit less, but it's still very acceptable. Better than what you're getting with Llama on Strix Halo, that's for sure.
These are small dense models,meaning Qwen, have gotten capable and fast. And there's been a lot of progress in the area. So sticking with Llama you are not taking advantage of the hardware you have. And yeah, for Strix Halo, you should just look into halogen and you shouldn't be just using 96 gigs for the VRAM. You should give it most of the VRAM to the inference and connect to it from your laptop or something. People ar egetting 1000+ pp with qwen 3.8 Next Flash
Your hardware can do way more than 64k tokens context window, can't it? And with Ollama it's very easy, superficially you just drag the slider.
I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now.
So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.
Yes, please!
Why are you using Ollama? Just use llama.cpp
I had hoped to get some new information out of this topic, but unfortunately found the same local "dead-ends" that I explored myself.
It unfortunately feels like we will be stuck waiting for a burst bubble before local hardware can be reasonably acquired for personal LLM usage.
The main gotcha for local models is insane hardware requirements.
Even for $10K you get mediocre performance.
I think this is worth revisiting once we get some solid 3rd party numbers from the new mac studio ultras, which admittedly are a bit over $10k with a 2tb ssd + 256gb ram. I think I'd be seriously considering it if I had a $200/month subscription of some kind.
This situation has improved quite a bit recently, Qwen Flash Next will run on a $4000 PC and can reliably implement small features on its own (feels comparable to Opus 4.5). It's a bit slow but pretty effective.
Not anymore. Qwen 3.8 27B is so great that 5k investment gives you excellent speed and quality.
Self-hosted Ollama is the best.