Change the behaviour of decode according to Unicode standard recommendation.
As a bonus dealing with partly broken encoding in the text becomes palatable which is hardly possible with the current behaviour.
The relevant section of the standard:
5.22 Best Practice for U+FFFD Substitution
When converting text from one character encoding to another, a conversion algorithm may
encounter unconvertible code units. This is most commonly caused by some sort of corruption
of the source data, so that it does not correctly follow the specification for that
character encoding. Examples include dropping a byte in a multibyte encoding such as
Shift-JIS, improper concatenation of strings, a mismatch between an encoding declaration
and actual encoding of text, use of non-shortest form for UTF-8, and so on.
...
Whenever an unconvertible offset is reached during conversion of a code
unit sequence:
1. The maximal subpart at that offset should be replaced by a single
U+FFFD.
2. The conversion should proceed at the offset immediately after the maximal
subpart.
---
Comment #1 by bearophile_hugs — 2014-02-08T14:43:03Z
Is it possible to generalize this to layz ranges? Currently this can't be nothrow. Can it become nothrow using your suggestion?
void main() nothrow {
string txt = "hello";
foreach (dchar c; txt) {}
}
Currently it gives:
test.d(3): Error: '_aApplycd1' is not nothrow
test.d(1): Error: function 'D main' is nothrow yet may throw
Comment #2 by issues.dlang — 2014-02-08T16:12:39Z
> Is it possible to generalize this to layz ranges?
I should point out that that code doesn't use any range-based functions at all. However, if decode and stride are both altered to be nothrow, then any function which was not inferred as nothrow because of them would then be nothrow (assuming that template inference was working correctly, which is not entirely the case right now - it does particularly poorly with Voldemort types). So, there's a good chance that a lot of range-based string stuff would become nothrow.
As for your example, that uses pure druntime stuff, as it's a string with foreach. And that code will need to be updated as well as std.utf.decode and std.utf.stride. However, theoretically, it will be in the same boat and be able to become nothrow, which would theoretically make your example nothrow, but I'd have to go digging through the code in druntime to see what all it does in order to be sure. I expect that it could be nothrow, but there might be something unexpected that it's doing that prevents it.