This paper seems dubious, because it flies in the face of what the reft/pyreft paper is showing (you can use 0.0001% of the parameters trained for 100 epochs to personalize on a small dataset):

https://github.com/stanfordnlp/pyreft

https://arxiv.org/abs/2404.03592

Note that the OP paper is not peer reviewed yet, and while the one I linked isn't either, it has Christopher Manning (yes, the one you know from youtube), the head of AI at Stanford, as a co-author.

In general, I think that Lora and especially reft should be more resistant to catastrophic forgetting due to them literally not impacting most of the model.

The Stable Diffusion community has literally tens of thousands of lora's that don't cripple a model at small rank.