Show HN: LLM Attention Visualization

This is great, thank you. I have to teach this stuff on Friday so perfect timing. It's hard to explain the attention mechanism in a way that becomes intuitive because the weighting scheme does not help much with the intuition. Having a visualization like this helps a lot. Don't move that page please since I'll link to it!

This is great, I've read multiple books and watched videos about the attention mechanism. Now that I understand it, this is the clearest example I've seen on how attention works.

neat, combining info from two phrases is hard to see without such a tool.

are you worried later-layer attention gets drowned out by earlier layers just because there are more of them contributing to the sum?

I like the visualisation. Pretty cool

I don't know much about LLMs but does that mean you have N^2 computation with the context size since every token needs to track how it relates to every other token?

I highly question this simplistic idea of high vector magnitude = high influence.

I am curious what's the actual formula.

I mean, there so many headers and layers, it is tricky to make a choice that will resonate with our intuition . Is it some weighted average? Or maybe ablation test?

[dead]

[dead]