Bug 24524 – Very slow process fork if RLIMIT_NOFILE is too high

In `std.process`, the `spawnProcessPosix` function, used by all functions that start a process, has a flag `Config.Flags.inheritFDs`. If set to false, it reads the maximum limit of files that can be opened by a process, `RFILE_NOFILE`, and then creates a structure for each one and finally iterates through them. Normally, the `RFILE_NOFILE` limit is set to 1024 or anyway a relatively low number. The hard limit depends on the system; in some, it is 2^20, in others even 2^30. When starting a container with Docker, the limit is raised to the maximum allowed, and this inevitably slows down the process visibly (because it has to allocate 2^20 (or 2^30) elements and then iterate through them. An user reported that on a system with 2^30 as the hard limit, it becomes literally unusable. There would be a faster way to do this by reading the open file descriptors from `/dev/fd` with the code below, but we are inside a `@nogc` block, so another solution must be found. ` import std.file : exists, isDir; if (exists("/dev/fd") && isDir("/dev/fd")) { import std.file : dirEntries, SpanMode; import std.path : baseName; import std.algorithm : map, filter; import std.array : array; import std.string : indexOfNeither; import std.conv : to; // List all file descriptors except stdin, stdout, stderr, and the pipe auto fds = dirEntries("/dev/fd", SpanMode.shallow) .map!(d => d.name.baseName) .filter!(d => d.indexOfNeither("0123456789") < 0) .map!(d => d.to!int) .filter!(d => d >=3 && d != forkPipeOut) .array; // Copy to avoid closing while iterating // Close only the file descriptors that are open. foreach(fd; fds) close(fd); } `

@trikko updated dlang/phobos pull request #8990 "Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high" fixing this issue: - Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high https://github.com/dlang/phobos/pull/8990

@schveiguy created dlang/phobos pull request #9077 "Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high" fixing this issue: - Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high https://github.com/dlang/phobos/pull/9077

dlang/phobos pull request #9077 "Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high" was merged into master: - 3c81b1ac8514ac17bef4375768eb9a0448bde9ac by Steven Schveighoffer: Fix bugzilla 24524: Very slow process fork if RLIMIT_NOFILE is too high https://github.com/dlang/phobos/pull/9077

THIS ISSUE HAS BEEN MOVED TO GITHUB https://github.com/dlang/phobos/issues/10548 DO NOT COMMENT HERE ANYMORE, NOBODY WILL SEE IT, THIS ISSUE HAS BEEN MOVED TO GITHUB