Internally I don’t think anyone knows.

I read an article once that talk about this the first few layers are like a translation layers then middle is thinking and the output layers translate it back.

It’s pretty interesting actually, you can cut out the middle layers and stack them back to back and that increases the models performance without much memory increase.

If you want to try it out you can get a LLM with no tools and pass it base64 and it will understand what’s in the message