I hope there will be an update where when my RAM gets full my PC doesn't freeze and becomes unusable... I remember that Linux and Windows do this in different ways and Windows doesn't have the problem.
Its not enabled by default, but enabling MGLRU setting /sys/kernel/mm/lru_gen/min_ttl_ms to 70 has entirely resolved the issue for me. It _does_ usually pick firefox or vscode to kill, but my system remains interactive, with no need for earlyoom.
Me too, but you can improve the behavior yourself too. It's been an issue with desktop linux and the default settings for a long time. You could maybe tune OOM killer settings, but another option is to use a package like earlyoom: https://man.archlinux.org/man/earlyoom.1.en
One of the biggest reason I migrated to Codex. I'd like to see each agentic CLI process as a simple building block, and having 100 claude session means I sometimes run out of memory (and I have 256GB of them). Codex never have this problem.
The communication is more direct too with GPT5.6 family of LLM. No more belt and suspenders.
Gosh, 7.2 literally just dropped with a bunch of awesome of performance/gaming related improvements (large folios, cache-aware scheduling, improved MGLRU reclaiming, Fair GPU Scheduler etc)... and I already can't wait for 7.3 to come out.
Meanwhile in the Windows world, users hate updates... Like I genuinely can't think of a single instance that made users exclaim, "oh boy I just can't wait for the next Patch Tuesday!".
I don't use windows too often, but I remember I was happy with an update a year or two ago which allows changing the volume by scrolling on the tray icon.
Now, of course, it still doesn't work with a touchpad, but baby steps, right?
I might be in the minority here, but some of the latest Windows updates that dropped in the Insider channel are really cool, and I'm excited to see those coming to main.
They seem to be focused on performance improvements because of the MacBook Neo pressure and RAM crisis, but right now, I'm mostly excited for the right-click menu and taskbar improvements. You can already do this with third-party software, but it's not the best experience unfortunately.
Are they finally going to fix the issue of having introduced a second right click menu that forces me to click “show more options” literally every single time I want to do anything?
Sadly I’m in a locked down corporate environment, so I can’t edit the registry. IT even closed my ticket when I asked for them to make the edit on my behalf.
I don’t use Windows on any personal machines, so there’s no way for this to benefit me. I hope it helps others who are frustrated by this though!
What, you don't find it charming that they shoehorned in the old right click menu without even changing its appearance to match the rest of Windows 11? It really shows how much thought Microsoft is putting into Windows now.
Fair point, the old school service packs and hotfix rollups were cool. But I think that sort of enthusiasm towards updates ended with XP, at least it did for me, because I switched to Linux permanently after Windows 7 came out.
XP was indeed when I stopped caring about Windows updates, SP2 was a huge deal - after that they where either "eh" or actively things I didn't want.
To be honest I think the reason I stopped care was more that I started using Linux as an OS in the 90's and over time more and more of my computing life was on Linux (except gaming) so by 2004 (or a little earlier) Windows was just for games.
Being able to preview media files directly in the preview pane of explorer was mind blowing to me. This feature alone made me love ME, let alone the interface customisation it allowed.
Any improvements with fractional scaling? Can anyone please link to any material reg the work being done for improving fractional scaling in Linux; it would be great if it could match the rendering quality of macOS.
IME it works fine on Intel iGPUs, too. I even use multiple ratios with no issue (constant 125% on the laptop, 100% on an external screen and 200% on a different screen at work).
Contrary to Windows, this also works well when hotplugging screens. Menus don't end up a blurry mess, and apps seem to work fine. On Windows, even some 1st-party apps are borked (task manager comes to mind).
Phoronix.com is my go to place for good news. It always gives me a sense of progress and makes me feel humble since so many smart people share their work
I use arch based linux. This year updates have broken video decoding twice. Font rendering once (still not fixed without manually deleting files.). And broken the whole system due to there not being a mechanism that makes sure all QT libraries upate at the same time once or twice. That is all one system.
Meanwhile 4 separate windows pcs have not had problems with updates for at least 6 years
It's not fair comparing Arch and Windows as they are two very different types of OSes. Arch is bleeding edge and it is completely normal and expected to be prone to breakages. It's not meant for the average PC user, it's meant for experienced users who're comfortable with troubleshooting technical issues, reporting bugs, and know and are happy to downgrade packages/rollback to previous snapshots etc. On the upside, Arch also gets fixes before anyone else.
If you want a stable Linux experience though, pick something like Fedora, or even better, pick an immutable distro like Aurora or Bazzite, if you really, really care about stability.
Great article! I share the same hunch as the author does; when allocating memory ultimately the application itself is in the best position to inform the kernel about the desired stickiness to VRAM. The best a kernel can do is guessing.
As a side note, it strikes me how much we owe to young trans people for low level performance engineering.
Of course. These are the people that the industry needs who understands proper performance-level engineering.
The commits they authored in [0] match and all of what they have mentioned in the article and this is how you know the author knows exactly what they are talking about and have explained it well.
Puts all the vibe-coders, and the so-called "AI Engineers" to shame. I know that if I ask about the basic system design of amdgpu and drm (direct rendering manager) infrastructure in Linux, a small handful will be able to explain it without using AI or googling.
This post focuses on performance, but what about crashing due to apps seeing no RAM available to allocate once VRAM is filled up?
I remember having this problem when I used an NVIDIA 750 Ti with 2 GB of memory. Just opening a few Firefox or Ghostty terminals (both are GPU accelerated) would result in those apps crashing or not being able to render their contents. Sometimes the compositor would crash or become unstable too (kwin, niri, etc.). I had to reboot every few hours. Complete system instability.
This problem only happened with Wayland, when I switched to X windows these problems went away.
Since then I switched to an AMD RX 480 (8 GB) GPU and never had an issue here with Wayland, if GPU memory gets full, system memory gets seamlessly used instead. It felt like with the NVIDIA card, it never allocated back to system memory if no VRAM was available. This was only a problem on Linux since that same NVIDIA card was fine in Windows for 10+ years with zero issues related to instability around VRAM allocation.
Tons of people reported the same issue on Linux on NVIDIA's forums for years.
Just search KWin Nvidia memory leak or Wayland Nvidia memory leak or kwin wayland Nvidia memory leak. It’s constantly issues and I remain convinced some intermixing of the two + monitors is at fault.
It's got to be an edge case for Wayland to be used with a 2GB VRAM card? Targeting recent cards and letting people using older hardware install x-windows, or whatever, seems sensible.
You say (in your blog post) "Linux" didn't work well so you reverted to "Windows", but as is so often the case you buried the distro/version info. You don't seem to have been using Win11 (later you mention Win7 & Win10Pro) ... So it's cutting edge software didn't work on my 2014 system, but older software did. The Linux/Windows aspects aren't particularly relevant AFAICT.
Nvidia refused for many years to properly support Linux, whilst AMD have supported it. So no surprise if a Nvidia card didn't work well/stably but an AMD one did.
Your last sentence sums it all: AMD is the only serious option on Linux. Not perfect, but far better than Nvidia, thanks to their open drivers that anyone (and especially Valve) can improve.
I gave myself 32GB of swap since SSDs are quite fast and the pages sent to disk seem to intelligently be rarely referenced ones. Feels amazing to run 4 VMs and 12 Claude Code instances on my mid tier hardware
Can you explain a little more please. I thought that graphics processing would revert to CPU if the GPU is maxed out and so swap might ultimately help? Is there no general GPU-swap mechanism? If CPU threads are competing for GPU resource does the GPU getting paged-out?
I suppose this could be an historic thing, GPUs can have a type of DMA now, and disk access is much faster too, so paging perhaps makes sense when it didn't before?
I worked on a PC/PS4/Xbox One game once. I wish I remembered more details, but at some point when looking at memory usage I found a 2048x2048 (or something like that) texture for something on a character. This texture was never anywhere near occupying the full screen. It got downsized once I told one of the artists, but I wouldn't be surprised if some games are inadvertently shipping with that kind of waste.
The release version of City Skylines 2 had enormous issues due to this problem. Some tiny details, like a pile of wood behind a shed, had ridiculous amount of pixels. I do believe that should be pretty much fixed in that title nowadays, however.
I doubt it makes much of a difference, and you can always manually manage what data lives in the GPU when if you 100% have to overcommit. Games have a much larger and more diverse set of objects in the VRAM, and their usage is less predictable, so manual scheduling of the memory is infeasible typically.
If you have model layers/experts that reside in CPU RAM, it's generally better to do that part of inference on the CPU than pay the cost of shipping them over to the GPU. LLM inference is generally bottlenecked by data-movement, not compute. There may be a limited exception for prefill or perhaps decode of very wide batches, where shuffling the data around may be justified.
Great article. I find that I learn something every time I read a post about linux kernel work.
I guess an LRU with priority would handle VRAM for games pretty decently without going getting too application specific.
What about VRAM to Disk specifically NVME, would direct to disk be feasible for large workloads, I know it is used for streaming in assets directly via. PCIE, but i wonder how the performance would be on compute workloads running with NVME as a swap for GPU VRAM.
Such great work!
Happens quiet regularly I hit a game just realizing seconds later I haven't shut down my local LLM yet. At least punishment will be less harsh. Lets see, maybe some games in window mode will just work fine.
This is a nice blog and it makes sense to me now. As a gamer and linux user myself, I've previously had to do tweaks and go-arounds without really understanding what was going on behind the scenes. :)
Great writeup, gpuvis looks particularly interesting and glad the kernel is providing tracepoints for performance events.
> Not only does the display hardware like scanned-out images to be in VRAM, it also completely skips past the GPU’s virtual memory architecture and works with physical addresses exclusively.
Well there's your problem. Only so smart your memory management can be when you have to pay the cost of doing it manually. Although presumably this only applies to a small fraction of the VRAM?
> Although presumably this only applies to a small fraction of the VRAM?
They did mention they saw 4GiB of eviction for a single 32MiB scan out image.
So while I would call the image allocation small, it seems to cause an avalanche of evictions. Amplified by the fact that each frame has one of these images, though I expect subsequent frames might have a better chance of already fitting into evicted space.
What I don't exactly understand is: doesn't it make sense to always reserve the contiguous physical memory for this case and not allow anything else to be put in it?
Or alternatively, instead of evicting it all, can you move the data around in physical memory while updating the page table, so that you can clear a large enough continuous block of physical memory?
I hope there will be an update where when my RAM gets full my PC doesn't freeze and becomes unusable... I remember that Linux and Windows do this in different ways and Windows doesn't have the problem.
Its not enabled by default, but enabling MGLRU setting /sys/kernel/mm/lru_gen/min_ttl_ms to 70 has entirely resolved the issue for me. It _does_ usually pick firefox or vscode to kill, but my system remains interactive, with no need for earlyoom.
Docs https://docs.kernel.org/admin-guide/mm/multigen_lru.html#thr...
Me too, but you can improve the behavior yourself too. It's been an issue with desktop linux and the default settings for a long time. You could maybe tune OOM killer settings, but another option is to use a package like earlyoom: https://man.archlinux.org/man/earlyoom.1.en
Windows absolutely does have this problem.
macOS too sadly. by the time you realise there's a problem the machine is already comatose and unresponsive
all these electron apps and invisible virtual machines (looking at you Claude) really don't help.
One of the biggest reason I migrated to Codex. I'd like to see each agentic CLI process as a simple building block, and having 100 claude session means I sometimes run out of memory (and I have 256GB of them). Codex never have this problem.
The communication is more direct too with GPT5.6 family of LLM. No more belt and suspenders.
zswap and systemd-oomd/earlyoom are your friends.
earlyoom is what you're looking for. it'll crash the process instead of hang
Gosh, 7.2 literally just dropped with a bunch of awesome of performance/gaming related improvements (large folios, cache-aware scheduling, improved MGLRU reclaiming, Fair GPU Scheduler etc)... and I already can't wait for 7.3 to come out.
Meanwhile in the Windows world, users hate updates... Like I genuinely can't think of a single instance that made users exclaim, "oh boy I just can't wait for the next Patch Tuesday!".
[1] Linux 7.2 Reverts DRM Scheduler Change After Serious GPU Regressions: https://www.linuxjournal.com/content/linux-72-reverts-drm-sc...
Ah, didn't know that was reverted. Cheers for the link.
I personally am very excited for the upcoming Windows 11 update which will allow users to move the taskbar to any edge of their screen.
I don't use windows too often, but I remember I was happy with an update a year or two ago which allows changing the volume by scrolling on the tray icon.
Now, of course, it still doesn't work with a touchpad, but baby steps, right?
Ah it'll be just like 2015 again
Wasn’t this in win 95?
Yep and it will be glorious.
I cant wait for the number one gripe of all my users to finally be catered for.
I might be in the minority here, but some of the latest Windows updates that dropped in the Insider channel are really cool, and I'm excited to see those coming to main.
Do you have a link for a list of these changes. I am curious.
Sure. The Insider blog tends to be pretty up-to-date: https://blogs.windows.com/windows-insider/
They seem to be focused on performance improvements because of the MacBook Neo pressure and RAM crisis, but right now, I'm mostly excited for the right-click menu and taskbar improvements. You can already do this with third-party software, but it's not the best experience unfortunately.
Are they finally going to fix the issue of having introduced a second right click menu that forces me to click “show more options” literally every single time I want to do anything?
Run (as Administrator) in a terminal
Should restore the old right click menu (it did last time I used it at any rate).So much of windows 11 configuration is removing things you never asked for and putting things back.
Source: https://learn.microsoft.com/en-us/answers/questions/2287432/...
Sadly I’m in a locked down corporate environment, so I can’t edit the registry. IT even closed my ticket when I asked for them to make the edit on my behalf.
I don’t use Windows on any personal machines, so there’s no way for this to benefit me. I hope it helps others who are frustrated by this though!
Lifesaver thankyou :)
What, you don't find it charming that they shoehorned in the old right click menu without even changing its appearance to match the rest of Windows 11? It really shows how much thought Microsoft is putting into Windows now.
Didn't we eagerly await on updates in the good old days of Windows?
Fair point, the old school service packs and hotfix rollups were cool. But I think that sort of enthusiasm towards updates ended with XP, at least it did for me, because I switched to Linux permanently after Windows 7 came out.
XP was indeed when I stopped caring about Windows updates, SP2 was a huge deal - after that they where either "eh" or actively things I didn't want.
To be honest I think the reason I stopped care was more that I started using Linux as an OS in the 90's and over time more and more of my computing life was on Linux (except gaming) so by 2004 (or a little earlier) Windows was just for games.
The only Windows update that made me “eagerly await” is the Windows XP SP2.
And what an update it was.
I still remember how Windows ME looked at first, the feeling of something better. That didn't last long
Being able to preview media files directly in the preview pane of explorer was mind blowing to me. This feature alone made me love ME, let alone the interface customisation it allowed.
Windows 7 was pretty neat.
I never did. I did wait eagerly for new .NET releases, though.
....No. And those were not good days, unless your definition is fun is reinstalling OS
Back when updates were add on packs like Plus! they were very exciting
https://en.wikipedia.org/wiki/Microsoft_Plus!
Any improvements with fractional scaling? Can anyone please link to any material reg the work being done for improving fractional scaling in Linux; it would be great if it could match the rendering quality of macOS.
What's your setup like? I'm on AMD+Wayland+KDE, no complaints with fractional scaling on my ASUS ProArt PX13 (255 PPI screen).
IME it works fine on Intel iGPUs, too. I even use multiple ratios with no issue (constant 125% on the laptop, 100% on an external screen and 200% on a different screen at work).
Contrary to Windows, this also works well when hotplugging screens. Menus don't end up a blurry mess, and apps seem to work fine. On Windows, even some 1st-party apps are borked (task manager comes to mind).
Phoronix.com is my go to place for good news. It always gives me a sense of progress and makes me feel humble since so many smart people share their work
I use arch based linux. This year updates have broken video decoding twice. Font rendering once (still not fixed without manually deleting files.). And broken the whole system due to there not being a mechanism that makes sure all QT libraries upate at the same time once or twice. That is all one system.
Meanwhile 4 separate windows pcs have not had problems with updates for at least 6 years
It's not fair comparing Arch and Windows as they are two very different types of OSes. Arch is bleeding edge and it is completely normal and expected to be prone to breakages. It's not meant for the average PC user, it's meant for experienced users who're comfortable with troubleshooting technical issues, reporting bugs, and know and are happy to downgrade packages/rollback to previous snapshots etc. On the upside, Arch also gets fixes before anyone else.
If you want a stable Linux experience though, pick something like Fedora, or even better, pick an immutable distro like Aurora or Bazzite, if you really, really care about stability.
That's an arch issue, not a Linux issue.
Great article! I share the same hunch as the author does; when allocating memory ultimately the application itself is in the best position to inform the kernel about the desired stickiness to VRAM. The best a kernel can do is guessing.
As a side note, it strikes me how much we owe to young trans people for low level performance engineering.
Of course. These are the people that the industry needs who understands proper performance-level engineering.
The commits they authored in [0] match and all of what they have mentioned in the article and this is how you know the author knows exactly what they are talking about and have explained it well.
Puts all the vibe-coders, and the so-called "AI Engineers" to shame. I know that if I ask about the basic system design of amdgpu and drm (direct rendering manager) infrastructure in Linux, a small handful will be able to explain it without using AI or googling.
[0] https://gitlab.freedesktop.org/pixelcluster/kernel/-/commits...
They'll get old soon enough.
This post focuses on performance, but what about crashing due to apps seeing no RAM available to allocate once VRAM is filled up?
I remember having this problem when I used an NVIDIA 750 Ti with 2 GB of memory. Just opening a few Firefox or Ghostty terminals (both are GPU accelerated) would result in those apps crashing or not being able to render their contents. Sometimes the compositor would crash or become unstable too (kwin, niri, etc.). I had to reboot every few hours. Complete system instability.
This problem only happened with Wayland, when I switched to X windows these problems went away.
Since then I switched to an AMD RX 480 (8 GB) GPU and never had an issue here with Wayland, if GPU memory gets full, system memory gets seamlessly used instead. It felt like with the NVIDIA card, it never allocated back to system memory if no VRAM was available. This was only a problem on Linux since that same NVIDIA card was fine in Windows for 10+ years with zero issues related to instability around VRAM allocation.
Tons of people reported the same issue on Linux on NVIDIA's forums for years.
I wrote a lot more details and recorded demo videos around 9 months ago here https://nickjanetakis.com/blog/gpu-memory-allocation-bugs-wi..., but since then a lot of these things have been resolved from switching over to AMD.
I have this problem with an 8gib 2080. Nvidia drivers have bugs with Wayland or Vulkan which is at least partially a memory leak.
Here’s one for example: https://forums.developer.nvidia.com/t/kwin-6-7-x-causes-mass...
Just search KWin Nvidia memory leak or Wayland Nvidia memory leak or kwin wayland Nvidia memory leak. It’s constantly issues and I remain convinced some intermixing of the two + monitors is at fault.
It's got to be an edge case for Wayland to be used with a 2GB VRAM card? Targeting recent cards and letting people using older hardware install x-windows, or whatever, seems sensible.
You say (in your blog post) "Linux" didn't work well so you reverted to "Windows", but as is so often the case you buried the distro/version info. You don't seem to have been using Win11 (later you mention Win7 & Win10Pro) ... So it's cutting edge software didn't work on my 2014 system, but older software did. The Linux/Windows aspects aren't particularly relevant AFAICT.
Nvidia refused for many years to properly support Linux, whilst AMD have supported it. So no surprise if a Nvidia card didn't work well/stably but an AMD one did.
Your last sentence sums it all: AMD is the only serious option on Linux. Not perfect, but far better than Nvidia, thanks to their open drivers that anyone (and especially Valve) can improve.
I gave myself 32GB of swap since SSDs are quite fast and the pages sent to disk seem to intelligently be rarely referenced ones. Feels amazing to run 4 VMs and 12 Claude Code instances on my mid tier hardware
Swap doesn't help in this case.
It's GPU drivers freaking out when they run out of VRAM, but it might only be specific to NVIDIA cards since AMD seems to handle this better.
But, if it's driver specific that means it could be fixed at the kernel level I suppose, especially since Windows handles it seamlessly?
Can you explain a little more please. I thought that graphics processing would revert to CPU if the GPU is maxed out and so swap might ultimately help? Is there no general GPU-swap mechanism? If CPU threads are competing for GPU resource does the GPU getting paged-out?
I suppose this could be an historic thing, GPUs can have a type of DMA now, and disk access is much faster too, so paging perhaps makes sense when it didn't before?
I'm mostly excited for the VRAM savings from apps and games which have loads of unused or super rarely used textures.
Game developers often aren't super careful with this stuff, and there could be literal gigabytes of data which isn't used at all.
I worked on a PC/PS4/Xbox One game once. I wish I remembered more details, but at some point when looking at memory usage I found a 2048x2048 (or something like that) texture for something on a character. This texture was never anywhere near occupying the full screen. It got downsized once I told one of the artists, but I wouldn't be surprised if some games are inadvertently shipping with that kind of waste.
The release version of City Skylines 2 had enormous issues due to this problem. Some tiny details, like a pile of wood behind a shed, had ridiculous amount of pixels. I do believe that should be pretty much fixed in that title nowadays, however.
wasn't it rendering every pedestrian's teeth in detail as well? hilarious oversight
Are there any people out there doing this optimisation post-release for existing games ?
Well written and very informative. I am glad we have these enthusiastic people around for Linux kernel development!
Enthusiasm seems proportional to the number of exclamation marks (28)! (:
I'm sure Linus will insult them into indifference
As proves the fact that this just got merged. :roll_eyes:
Downvoted for slating lord Linus... Fuck him and the lot of you
meh.
I'll be the one to ask the obvious question:
What does this mean for compute workloads? Specifically, LLM inference.
Does it mean anything at all, or is this purely a games-thing?
I doubt it makes much of a difference, and you can always manually manage what data lives in the GPU when if you 100% have to overcommit. Games have a much larger and more diverse set of objects in the VRAM, and their usage is less predictable, so manual scheduling of the memory is infeasible typically.
If you have model layers/experts that reside in CPU RAM, it's generally better to do that part of inference on the CPU than pay the cost of shipping them over to the GPU. LLM inference is generally bottlenecked by data-movement, not compute. There may be a limited exception for prefill or perhaps decode of very wide batches, where shuffling the data around may be justified.
Great article. I find that I learn something every time I read a post about linux kernel work.
I guess an LRU with priority would handle VRAM for games pretty decently without going getting too application specific.
What about VRAM to Disk specifically NVME, would direct to disk be feasible for large workloads, I know it is used for streaming in assets directly via. PCIE, but i wonder how the performance would be on compute workloads running with NVME as a swap for GPU VRAM.
> but i wonder how the performance would be on compute workloads running with NVME as a swap for GPU VRAM.
4x as slow in absolute best case, NVMe drives have 4 PCIe lanes usually
Such great work! Happens quiet regularly I hit a game just realizing seconds later I haven't shut down my local LLM yet. At least punishment will be less harsh. Lets see, maybe some games in window mode will just work fine.
This is a nice blog and it makes sense to me now. As a gamer and linux user myself, I've previously had to do tweaks and go-arounds without really understanding what was going on behind the scenes. :)
Great writeup, gpuvis looks particularly interesting and glad the kernel is providing tracepoints for performance events.
> Not only does the display hardware like scanned-out images to be in VRAM, it also completely skips past the GPU’s virtual memory architecture and works with physical addresses exclusively.
Well there's your problem. Only so smart your memory management can be when you have to pay the cost of doing it manually. Although presumably this only applies to a small fraction of the VRAM?
> Although presumably this only applies to a small fraction of the VRAM?
They did mention they saw 4GiB of eviction for a single 32MiB scan out image.
So while I would call the image allocation small, it seems to cause an avalanche of evictions. Amplified by the fact that each frame has one of these images, though I expect subsequent frames might have a better chance of already fitting into evicted space.
What I don't exactly understand is: doesn't it make sense to always reserve the contiguous physical memory for this case and not allow anything else to be put in it?
Or alternatively, instead of evicting it all, can you move the data around in physical memory while updating the page table, so that you can clear a large enough continuous block of physical memory?
Also Apple M3 support