> What I'd say instead is: If native speakers say something, then it's grammatically correct.

By your own logic, "literally" meaning "figuratively" is grammatically correct. Which just goes to show that your logic is wrong. Something can in fact be grammatically incorrect even when said by native speakers.