I leave Claude memory stuff turned off by default for the same reason. It inevitably ends up dragging in things that are unrelated and unimportant to the current task.