Modern OS exploit chains should give you a good sense of how far people can go. (Eg, phone OSes are relatively hardened.)

We’re not even at the “ASLR” level of protection for LLMs yet.