Bug 5904 – std.json parseString doesn't handle chars outside the BMP

According to RFC 4627, characters outside the Basic Multilingual Plane (ie. those that require more than two bytes to represent) are encoded as a surrogate pair in JSON strings. In effect, what you have to do is test whether a "\uXXXX" value is >= 0xD800 and <= 0xDBFF. If so, then the next value should be another "\uXXXX" character representing the low surrogate. To verify this, the value should be >= 0xDC00 and <= 0xDFFF. If it isn't, then skip the preceding "\uXXXX" value (the high surrogate) as invalid and decode the following "\uXXXX" value as a standalone Unicode code-point (the RFC is actually unclear on this point, but this seems the most reasonable failure mode). Assuming that you have a valid high and low surrogate, stick them into a wchar[2] and convert to UTF8.

Test case: ///////////// test.d ///////////// import std.json; void main() { string s = `"\uD834\uDD1E"`; auto j = parseJSON(s); assert(j.str == "\U0001D11E"); } //////////////////////////////////

Commit pushed to stable at https://github.com/dlang/phobos https://github.com/dlang/phobos/commit/b23e7a4107cc2eb3275e022cb46f7270e586ca29 Fix Issue 5904 - std.json parseString doesn't handle chars outside the BMP

Commit pushed to master at https://github.com/dlang/phobos https://github.com/dlang/phobos/commit/b23e7a4107cc2eb3275e022cb46f7270e586ca29 Fix Issue 5904 - std.json parseString doesn't handle chars outside the BMP

Commit pushed to dmd-cxx at https://github.com/dlang/phobos https://github.com/dlang/phobos/commit/b23e7a4107cc2eb3275e022cb46f7270e586ca29 Fix Issue 5904 - std.json parseString doesn't handle chars outside the BMP