logoalt Hacker News

n_plus_1_acc • today at 3:54 PM • 2 replies • view on HN

Interesting, thanks. How did you generate this list? And do you know if the lowercasing rules have ever changed between Unicode version?

Replies

Rendello • today at 7:21 PM

Lowercasing is not guaranteed to be stable between Unicode versions. If you're looking for stable string matching, use "case folding" instead. Instead of:

    lower(a) == lower(b)             // or,
    lower(needle) in lower(haystack)
Use:

    casefold(a) == casefold(b)             // or,
    casefold(needle) in casefold(haystack)
It's equivalent to lowercasing in almost all cases, but with a stability constraint between across Unicode editions. As per the Unicode stability policy [1]:

> Case folding is not the same as lowercasing, and a case-folded string is not necessarily lowercase. In particular, as of Unicode 8.0, Cherokee has become a bicameral script with the introduction of lowercase Cherokee letters, but Cherokee text case folds to the existing uppercase letters. This case folding behavior for Cherokee text is precisely to guarantee continued case folding stability.

Casing in Unicode can be tricky. The case mapping FAQ [2] is a good place to start. If you need more information, start scanning through the Core Spec and following the case-related links [3].

1. https://www.unicode.org/policies/stability_policy.html#:~:te...

2. https://www.unicode.org/faq/casemap_charprop.html#casemap

3. https://www.unicode.org/versions/Unicode17.0.0/core-spec/cha...

teddyh • today at 4:20 PM

Via <https://data.iana.org/TLD/tlds-alpha-by-domain.txt>, <https://www.unicode.org/Public/UNIDATA/UnicodeData.txt>, some Python and some regexps.

(I don’t know anything about Unicode case conversions.)