This isn't happening for me, if by "over time" you mean "several weeks" of heavy GPU use without a reboot. Consider the possibility that the root cause is not macOS.
By "over time" I mean that if I run local LLMs or play graphics-intensive games (like No Man's Sky's native Mac port) I can start seeing up to half-second (or longer) delays in switching apps, starting videos and much more with less than 5 days of uptime. Heavy enough usage will reproduce it within a few hours.
If I avoid doing those things, it can last much longer.
I have the 128GB/8TB M4 Max and I'm not even coming close to using up all or even most of the unified memory. I suspect it may be related to some form of fragmentation, though I have no way to know how.
By "over time" I mean that if I run local LLMs or play graphics-intensive games (like No Man's Sky's native Mac port) I can start seeing up to half-second (or longer) delays in switching apps, starting videos and much more with less than 5 days of uptime. Heavy enough usage will reproduce it within a few hours.
If I avoid doing those things, it can last much longer.
I have the 128GB/8TB M4 Max and I'm not even coming close to using up all or even most of the unified memory. I suspect it may be related to some form of fragmentation, though I have no way to know how.