I strongly agree. This is becoming a general principle along the lines of “code comments that describe what the code does are bad.”
Memory just ensures there is a fairly high level of drift and inaccuracy in your context. Unless YOU are manually curating it.
Its similar to wiki systems where your agent produces tons of documentation and decision records, etc. Its a bunch of assertions about your codebase that you aren’t vetting and which is at most 90% correct. Thats good enough for a feature or bug fix that you can validate with tests but for the love of god throw the details away. You don’t want to codify that. It’s polluting the context and gimping the model.