That means 7 is also a measure of bytes, just slightly more awkward. So it's roughly on par with 17.
For 5, the idea is that while you might want to iterate code points, the total number of code points is less useful than either grapheme count or byte count. I think that argument makes sense.
> That means 7 is also a measure of bytes, just slightly more awkward.
It's not a real measure of bytes though. It's the count of bytes in an encoding scheme that is (probably) neither what you use to communicate with the outside world nor what your language runtime uses. (And certainly it's no better than 5, since that's also a measure of bytes in a particular encoding).
Lots of systems use UTF-16 internally and externally. Counting bytes in UTF-16 is, on average, almost as useful as counting bytes in UTF-8.
I don't think just about anything communicates in UTF-32. 5 is basically just a codepoint count, and as such I don't think its usefulness rating should be between the byte counts.
> Lots of systems use UTF-16 internally and externally. Counting bytes in UTF-16 is, on average, almost as useful as counting bytes in UTF-8.
Not my experience at all. The article points out that even languages that are committed to an UTF-16 interface prefer to use other internal storage representations, and I can't remember the last time I saw it used in a transfer format.
I hate UTF-16 and the systems that use it with a passion, but...
Windows and Java (and Javascript) adopted unicode at a time when it was thought that 64k code points would be enough for everyone. Then they prioritized backwards compatibility over anything else. Most of us have benefited from their insistence on backwards compatibility in some form or the other, so I'm really not in a position to complain about it :-/
That said, IMHO any "length" property (as opposed to `codepoints` or `bytes`) on a UTF-16 string should definitely be deprecated.
Windows, Java, C#, javascript, a surprising number of XML documents (though less so as time marches on thankfully), ICU I think uses UTF-16 internally (for the same historical reasons as the other 4), JOLIET file names are UCS2, some phones interpret β16-bitβ SMS as UTF-16 (the spec says UCS2).
> and BOTH of those are insane for sticking to it
They donβt really have much of a choice because they exposed those semantics as part of the string interface (or for Windows the interaction is slow low level it canβt be hidden), they have performance guarantees and behaviours which matches that.
Itβs also why Python uses UTF-32, and went through the entire PEP-393 / FS complication to try and stop blowing up memory left and right: the core team considered that switching strings to UTF8 was a bridge too far.
There are approximate solutions, but they come with their own costs and complications (e.g. pypy uses UTF8 strings with lazily constructed indices to emulate UTF-32 strings).
I'm not a Windows based programmer, but couldn't they leave the old API's in place, but make UTF-8 safe versions available for everyone and switch to that... E.G. with Win 11?
You can set the system codepage to CP_UTF8 since Win 10, I guess, although IIRC it still doesn't work for input. But a) there is a lot of programs using A() functions that don't expect that and break in subtle ways, e.g. DBCS-encoding-aware programs suddenly break because they don't expect a codepoint to span for more than 2 bytes; b) most of the sanely written programs either use UTF-16 explicitly, or use UTF-8 internally and convert between UTF-8 and UTF-16 before/after calling W() functions.
The JavaScript language forces utf16 (whether or not v8 uses that representation under the hood). For instance if you want to substring the indexes you pass are for utf16 codepoints
I think that argument makes as much sense as saying that an engine is less useful than a car. And pretending that engine.weight should return the weight of the car.
For 5, the idea is that while you might want to iterate code points, the total number of code points is less useful than either grapheme count or byte count. I think that argument makes sense.