Well the embedding itself is in some latent space.

Wq projects it to the space of queries. I.E What questions is this token asking?

Wk projects it to the space of keys. I.E What questions does this token answer.

Wk projects it to the space of values. I.E What are those answers?

Of course this explanation is prescribed onto the matrixes after the fact.

You can in fact do weird stuff like construct weights so attention calculates least squares, or sorts numbers, or other weird constructions like a transformer that calculates gradient descent steps. It seems to be very flexible in terms of what functions on data it can encode.