alt Hacker NewsOops, I legitimately missed the second-to-last paragraph.
I still think there are better tests you could do. Ideally, you would choose a book that was published recently—after the model’s cut-off date—which is considered to be a good translation. But even something like The Girl With the Dragon Tattoo, which is not particularly new and by no means obscure, would be better than a famous work of literature like The Three Musketeers that has many translations.
Almost certainly correct, though I've noticed that these LLMs like to complain when you give it stuff that is still in copyright. The Three Musketeers is thoroughly public domain everywhere so in that sense it's a good test, but of course because it's public domain everywhere there are lots of translations to crib from so I acknowledge it's not a great test because the training data almost certainly contains a competent translation.
Even if Fable didn't have Ellsworth's translation, it certainly has the William Barrow translation, which would still get it like 80+% of the way there.
My wife speaks Spanish, I should get her to do some kind of comparison with a Spanish book that doesn't have English translations.