I don't think anyone is arguing that Kyber is purposefully backdoored. They are arguing that it (and basically every other lattice based method) has lost a minimum of ~50-100 bits of security in the past decade (and half of the stage 1 algorithms were broken entirely). The reason I can only give ~50-100 bits as the amount Kyber has lost is because attacks are progressing fast enough, and analysis of attacks is complicated enough that no one has actually published a reliable estimate of how strong Kyber is putting together all known attacks.
I have no knowledge of whether Kyber at this point is vulnerable given whatever private cryptanalysis the NSA definitely has done on it, but if Kyber is adopted now, it will definitely be in use 2 decades from now, and it's hard to believe that it won't be vulnerable/broken then (even with only publicly available information).
Source for this loss of security? I'm aware of the MATZOV work but you make it sound like there's a continuous and steady improvement in attacks and that is not my impression.
Lots of algorithms were broken, but so what? Things like Rainbow and SIKE are not at all based on the hardness of solving lattice problems.