alt Hacker NewsThe constant issue with these sorts of categorization efforts is that the outcome is entirely dependent on how the responses to "politically charged questions" are graded as left vs. right. You're mostly just examining a delta in biases between the model and the investigator.
Replies
Yes, but I think it is still a viable metric to some degree. I wondered about Gemini being dead center here. At first it was obvious that it was actively trained to give biased responses to anything controversial. It was deservedly made fun off because it tried to warp reality. I still don't trust it today, although that is pretty much true for any model.
The alternative is a High-Dimensional / Embedding-Like Approach where question responses aren't tied to fixed axes, but rather the full response set is treated as a point in latent space.
Then it's on the researcher to examine the clusters and assign labels. There's also not a nice mapping that's a-priori interpretable in low-dimensional pre-existing axes.
Probably only used in research than consumer websites, under more controlled conditions; there are very few public political tests doing this transparently
Your multimeter reads out voltage relative to the black terminal, it's your responsibility to find the ground plane.
Exactly. These models don’t hold coherent views. You can prime any of them to agree with any view.
The other issue is that it is going to depend very strongly on how you ask.
This is especially apparent in the 'worldview' sorting under the `bias` section, which lists the German FDP to be further right than the CDU (which is nonsense) and also barely registers the FDP as libertarian when they are a free speech, small government, personal responsibility and free market party. They also register "Die Linke" as Libertarian-Left, which could not be further from the truth. "Die Linke" barely has libertarian values at all, being pro state-governed economy, having an ultimate goal of democratic socialism and they're certainly big government. They're also leading a large deposession effort for large landlord companies. I'd honestly put them into "Auth-Left" territory.
So yeah. The bias is a bit nuts and you could reasonably accuse the study/report of misdirection/misinformation and plain fasehoods.
Fair; have tried to combat this issue in a few ways.
Each model's position is scored against outside political-science data (Chapel Hill Expert Survey for party positions, World Values Survey for where populations sit).
The stance coding is done by a separate model with a published prompt + a second model from a different lab re-scores a sample and we publish where the two disagree.
So not perfect but (as far as I can tell) one of the more defensible approaches.