Counting the characters in ONE CHARACTER remains an open problem in computer science.
* PHP's strrev reverses a string byte-wise, thereby corrupting UTF-8. 🤦
* For Python's encode() UTF-16 is neither UTF-16-LE nor UTF-16-BE. 🤨
* NUL-terminated C-Strings. 💀
* strlen() 👻
Do you know further string processing horrors?
@fluepke Well… once you know how to do it, it all makes perfect sense… 🤡
@alvar @fluepke For completeness:
use v5.36;
binmode STDOUT, ':encoding(UTF-8)';
my $flag = "\N{U+1F3F3}\N{U+FE0F}\N{U+200D}\N{U+26A7}\N{U+FE0F}";
my $codepoints = length $flag;
my $graphemes = () = $flag =~ /\X/g;
utf8::encode $flag;
my $bytes = length $flag;
say "codepoints: $codepoints";
say "graphemes: $graphemes";
say "bytes: $bytes";
Output:
codepoints: 5
graphemes: 1
bytes: 16
@fluepke In MySQL, when you ask for "utf8" encoding, you don't get UTF-8 encoding. You get "UTF-8, but not beyond U+FFFF".
If you want "UTF-8, like, really" you have to ask for "utf8mb4" encoding.
@fluepke Having worked in the JVM space for more than 20 years and seeing this, triggers extreme impostor syndrome. Maybe gardening or woodworking would have been a better choice.
@m303
Being an impostor in IT is easy, "unexpected behaviour, no one could have guessed" works often enough as excuse. But everyone can evaluate the output of other professions (the flower is dead or not, the chair is wobbly or not). Explain issues away with hand-waving is outside of IT more at home in, say, politics and lawyering.
(full disclosure: I work in IT, too. And share your sentiment, doing something with my hands is often enough a dream. But an unrealistic one, I fear)
@fluepke
Computer scientists be like:
Let Σ = {a, b, c} be an alphabet. A string is an element of Σⁿ. DONE. STRINGS SOLVED. QED. 👏
Engineers meanwhile:
Humanity has spent thousands of years developing hundreds of writing systems and countless ways to express meaning through text. Let's account for that!
Meanwhile, some guy writing C since 1978:
Wait, there's writing beyond ASCII? Sounds like a skill issue. If you need umlauts, you deserve the extra screening at US immigration!
@fluepke Japanese characters can be written in wide or narrow form, which have different Unicode encodings. the width depends on the device you're typing on, with narrow being common as a default on mobile soft-keyboards and wide being the typical default when writing on a desktop keyboard. as such, passwords written in CJK scripts may encode to different byte sequences depending on whether the user logs in via mobile or desktop, and therefore produce different hashes/keys.
@fluepke you have to run NFC normalisation over the string first to get a consistent byte sequence.
however! this is not actually sufficient on its own. another gotcha is that the normalisation result depends on the Unicode version. for long term stability (e.g. if you're using the password as an input to a KDF for bulk storage encryption) you need to pin the Unicode version alongside the hash. failing to do so many result in a user no longer being able to recover data.
@fluepke and since a new user password might contain grapheme clusters that only exist in newer Unicode versions, you either need to have a version upgrade path (no global version pin) or reject any password that contains an unrecognised cluster.
this is even more difficult if you're doing any kind of client-side / server-side agreement, especially across distinct platform and language stacks, because both sides need to be on the same Unicode version and be doing the same normalisation.
@fluepke also: if your threat model includes timing attacks, you need to do the Unicode cluster validity checks and NFC normalisation in constant time.
@fluepke also also: if you're doing any kind of nonrepudiation of string equality where malicious modification is in scope you also need to authenticate the Unicode version that was used, e.g. in the AD portion of an AEAD.
@gsuberland @fluepke A certain Japanese airline (and I bet it is not limited to them)even has the PIN requirement of "4 half-width numbers".
It struck me as odd when seeing this, and now I don't want to contemplate the implications...
@fluepke huh, my 🏳⚧ is just 10 bytes: f0 9f 8f b3 e2 80 8d e2 9a a7
As of Unicode 17.0:
Emoji variation sequences contain VS16 (U+FE0F) for emoji-style (with color) or VS15 (U+FE0E) for text style (monochrome). This use is deprecated; newer emoji versions of existing dingbat characters have a separate code point for the emoji form.
(wikipedia)
@noisytoot @fluepke @domi
Ahh wait, i see. Firefox renders the flag symbol with Noto Sans Symbols 2 font, and the transgender symbol with DejaVu Sans. If i explicitly set font-family from DevTools to "Noto Color Emoji", it doesn't split it and show as a colored trans flag emoji
@noisytoot @fluepke @domi
So that depends on how font preference is configured and what is available in the system.
But probably should also be fixed on Firefox's side, because it doesn't seem correct to choose different fonts for symbols and accidentally split a single emoji
@fluepke mysql utf8 is not full utf8 and only uses 3 bytes
So when you want to store stuff like emoji, it completely falls apart, unless you use utf8butforrealthistime
err, I mean
utf8mb4