Bug 11791 – std.file.write failed to write huge files

std.file.write failed to write files over 4GB (in my case 3.3GB). But std.stdio works well.

Would it be possible to provide more information? In what way does it fail (exception, seg fault, just stops writing at a certain point, etc.)

import std.file; void main() { auto data = new byte[2L^^31]; std.file.write("data", data); } std.file.FileException@std/file.d(391): data: Success ---------------- /tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void std.file.writeImpl(const(char[]), const(void[]), const(uint))+0xe1) [0x451fe1] /tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void std.file.write(const(char[]), const(void[]))+0x38) [0x451ef8] /tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(_Dmain+0x52) [0x44f242] /tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll().void __lambda1()+0x18) [0x44f990] /tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x44f8ea] /tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()+0x30) [0x44f950] /tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x44f8ea] /tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(_d_run_main+0x1a3) [0x44f86b] /tmp/.rdmd-1000/rdmd-test.d-60F3F2708DEB1FEF9505527614062FAE/test(main+0x17) [0x44f25f] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f73bff53de5] ---------------- [Finished in 18.5s with exit code 1] [cmd: ['rdmd', '/home/ilya/Desktop/test.d']] [dir: /home/ilya/Desktop] [path: /usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/ilya/[email protected]/system/ldc/bin/]

(In reply to comment #2) > import std.file; > > void main() { > auto data = new byte[2L^^31]; > std.file.write("data", data); > } > > std.file.FileException@std/file.d(391): data: Success works fine with 2L^^30

(In reply to comment #3) > (In reply to comment #2) > > import std.file; > > > > void main() { > > auto data = new byte[2L^^31]; > > std.file.write("data", data); > > } > > > > std.file.FileException@std/file.d(391): data: Success > > works fine with 2L^^30 Ok, so that line indicates that the call to "close" failed (returned non-zero), but didn't set errno, which is weird. One possibility is that you are running a 32-bit kernel, and write will only be able to write INT_MAX bytes, which would explain the error (although I would have hoped for it to set errno). Are you running a 32-bit kernel? I suppose an alternate question is: can you write a file like this using plain C functions, or is it a D specific issue? Phobos literally just forwards the call onto a few C functions (open file, write, close), so I suspect it is not an issue in Phobos.

(In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #2) > > > import std.file; > > > > > > void main() { > > > auto data = new byte[2L^^31]; > > > std.file.write("data", data); > > > } > > > > > > std.file.FileException@std/file.d(391): data: Success > > > > works fine with 2L^^30 > > Ok, so that line indicates that the call to "close" failed (returned non-zero), > but didn't set errno, which is weird. > > One possibility is that you are running a 32-bit kernel, and write will only be > able to write INT_MAX bytes, which would explain the error (although I would > have hoped for it to set errno). Are you running a 32-bit kernel? > > I suppose an alternate question is: can you write a file like this using plain > C functions, or is it a D specific issue? Phobos literally just forwards the > call onto a few C functions (open file, write, close), so I suspect it is not > an issue in Phobos. kubuntu 13.04 $ uname -a Linux Y 3.11.0-14-generic #21-Ubuntu SMP Tue Nov 12 17:04:55 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux D programs also are x86_64, but it is possible to use x86_32 C library in ubuntu. With std.stdio.File I can write an 6 GB files.

This is a known limitation of the write function. see http://man7.org/linux/man-pages/man2/write.2.html > on Linux, write() (and similar system calls) will transfer at most > 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes > actually transferred. (This is true on both 32-bit and 64-bit > systems.) If you want to write more you must use a stream and write several buffers or use append(). std.stdio.File works because it doesn't bulk-write.

> If you want to write more you must use a stream and write several buffers or use append(). std.stdio.File works because it doesn't bulk-write. Personally std.file could just write huge memory chunks in multiple write calls. There is no requirement for it to be single syscall.

Commits pushed to master at https://github.com/dlang/phobos https://github.com/dlang/phobos/commit/2ff9afaeeec6d0cbfb0640f21c73fe6a4c3a682d fix issue 11791 - std.file.write failed to write huge files https://github.com/dlang/phobos/commit/d0ef84f04a6cde73eceebea4ffc7f405f01fdc66 Merge pull request #4107 from BBasile/issue-11791 fix issue 11791 - std.file.write failed to write huge files