alt Hacker NewsUtf8 solved this completely. It works with any length unicode and on average takes up almost as little storage as ascii.
Utf16 is brain dead and an embarrassment
Replies
wvenable • today at 1:30 AM
Blame the Unicode consortium for not coming up UTF-8 first (or, really, at all). And for assuming that 65526 code points would be enough for everyone.
So many problems could be solved with a time machine.
➕ show 1 reply
It gets worse for UTF-16, Windows will let you name files using unpaired surrogates, now you have a filename that exists on your disk that cannot be represented in UTF-8 (nor compliant UTF-16 for that matter). Because of that, there's yet another encoding called WTF-8 that can represent the arbitrary invalid 16-bit values.