By "over time" I mean that if I run local LLMs or play graphics-intensive games (like No Man's Sky's native Mac port) I can start seeing up to half-second (or longer) delays in switching apps, starting videos and much more with less than 5 days of uptime. Heavy enough usage will reproduce it within a few hours.

If I avoid doing those things, it can last much longer.

I have the 128GB/8TB M4 Max and I'm not even coming close to using up all or even most of the unified memory. I suspect it may be related to some form of fragmentation, though I have no way to know how.