What thousands of conversations look like inside a language model

I’ve been playing around with mapping steam games + their reviews based on Qwen embeddings. I revisit this project every year around this time to see what improvements I can make to make a “better” map than last year (According to my own preferences). [0/1]

One thing I learned this year is that naive text mapping (embed a corpus of reviews with a Qwen embedding model, use umap to plot, then compare where things fall) primarily compares embedded text length more than anything else. What a review says affects the position less than how long the review is.

This was a little surprising to me. For whatever reason, I didn’t think the embedding would so strongly describe the length of the embedded passage. So most of this year’s time was spent fixing that artifact.

[0]: 2025’s map: https://storage.danieltperry.me/share/steam-map/

[1]: 2026’s map (still a work in progress, both the ui and the review processing are incomplete): https://storage.danieltperry.me/share/steam-map-refresh/

umap (https://umap-learn.readthedocs.io/en/latest/) or tsne (https://scikit-learn.org/stable/modules/generated/sklearn.ma...) might be an interesting alternative to pca for dimensionality reduction

The methodology feels either heavily undercooked to the point of being misleading, or heavily under explained. Why are we in 3D? At which point did dimensionality reduction hit? And if we're doing dimensionality reduction, does the "plane" really mean anything?

I wish there had been a third broad language style such as arabic although perhaps Qwen doesnt use arabic.

It made me think what if totally different knowledge bases could exist in a model where as my mental model was similar to OPs translation from pure knowledge source.

It also explains why rewriring prompts to another language sometimes lets you bypass restrictions.

I think the raw tokens embeddings shouldn't have been included in the PCA (past the first or second graph). Since they're disjoint from the conversation embeddings, you're essentially losing a dimension and compressing all the conversations into a plane. There may well have been much more obvious structure in the language embeddings that you'd be able to see.

The language comparison angle is interesting; a broader set of languages would make this even more informative.