Bug 13651 – Writing Unicode text with console code page 65001 (UTF-8) may fail

Created attachment 1447 Data for demo program //////////////// demo.d /////////////// import core.sys.windows.windows; import std.file; import std.stdio; void main() { SetConsoleOutputCP(65001); stdout.write(readText("data.txt")); } /////////////////////////////////////// data.txt attached. This crashes in different ways on Win32 and Win64. In both cases, fwrite completes partially (no error with DMC runtime, "broken pipe" with MSVC runtime).

BTW, I've tried reducing data.txt, and I think it's no coincidence that its size is 1 past 16384 (0x4000).

Is "broken pipe" error reported by MSVC runtime itself? The file interface to console stream in utf-8 encoding is a little tricky: the program shouldn't check the number of written bytes (it's probably messed by transcoding).

(In reply to Vladimir Panteleev from comment #1) > BTW, I've tried reducing data.txt, and I think it's no coincidence that its > size is 1 past 16384 (0x4000). You mean it works for smaller files? There's size limit for string written to console: https://connect.microsoft.com/VisualStudio/feedback/details/635230/

Yes. The program I posted throws an exception because fwrite returns a number (bytes written) smaller than the size of the data. According to its documentation, this indicates a write error. I've tried modifying std.stdio to retry incomplete fwrites with the remaining data segment, but although now no exception is thrown, the output is corrupted (some data is repeated).

I guess sometimes fwrite will return a number different from the "count" parameter, even though it wrote everything without an error?

I think, it applies to console only. By implementation, WriteFile checks if the file is console and calls WriteConsoleA, which can think in characters instead of bytes.

Indeed, docs for WriteConsole say it operates in terms of characters without definition of the character and indeed it reports the number of written characters, not bytes.

FYI there's a similar problem with the curl command line tool with chcp 65001. chcp 65001 curl https://issues.dlang.org/attachment.cgi?id=1447 curl: (23) Failed writing body (3019 != 8000)

Minimal reproducible test cast: import core.sys.windows.windows; import std.stdio; void main() { SetConsoleOutputCP(65001); stderr.writeln("あ"); } Which causes this exception: std.exception.ErrnoException@C:\D\dmd2\windows\bin\..\..\src\phobos\std\stdio.d(2889): Enforcement failed (No error) ---------------- 0x00402499 0x0040242E 0x00402328 0x00402263 0x00404275 0x0040410F 0x0040268F 0x760B343D in BaseThreadInitThunk 0x77039832 in RtlInitializeExceptionChain 0x77039805 in RtlInitializeExceptionChain Normal writeln without stderr or using stdout works.

*** Issue 14129 has been marked as a duplicate of this issue. ***

THIS ISSUE HAS BEEN MOVED TO GITHUB https://github.com/dlang/phobos/issues/9644 DO NOT COMMENT HERE ANYMORE, NOBODY WILL SEE IT, THIS ISSUE HAS BEEN MOVED TO GITHUB