alt Hacker Newsreplace they key-query-value mechanic by just dropping it while making the entire context the latent space.
What do you mean by this? Like concatenating all token embeddings into one large vector?
replace they key-query-value mechanic by just dropping it while making the entire context the latent space.
What do you mean by this? Like concatenating all token embeddings into one large vector?
Yes. Just have the entire context visible to everything, all the time.
It's just one way to increase the expressiveness of the whole network. In general one could say that coming up with ways to greatly increase the expressiveness of neural nets is not hard. Or it's hard because it blows up compute. Meanwhile the human mind demonstrates that you can drop compute by 5-6 orders of magnitude without losing expressiveness to save power.