Internally I don’t think anyone knows.

I read an article once that talk about this the first few layers are like a translation layers then middle is thinking and the output layers translate it back.

It’s pretty interesting actually, you can cut out the middle layers and stack them back to back and that increases the models performance without much memory increase.

If you want to try it out you can get a LLM with no tools and pass it base64 and it will understand what’s in the message

It seems a ton of the chain of thought tokens are English. It just processes weights, though, not tokens - maybe compute efficiency for using "cached" tokens/weights or it more efficiently learned to process English tokens by virtue of their greater weight in the corpus.

In Deepseek app you can see its reasoning tokens.

If I ask something in my native language (Korean), the model often switches into English in the middle and translate the answer into Korean.

But not sure about what happens below the token, in embedding space or per decoder layers.