Recursive self improvement is a bulk process that shapes the weights of the next model. The weights are still frozen. Each individual "agent" would need to able to durably update its own dispositions to get to what I'm talking about.

I'm not suggesting mathematics and language are the same thing, but I am saying (which shouldn't be controversial) that mathematics as a discipline exists within the medium of language, and that mathematical reasoning, discovery, and proof happen via people proposing, justifying, testing, and objecting to formulations in language. This isn't a metaphysical claim about whether math is real or captures inherent structures of the universe etc.