Bug 11282 – std.process: add capability for two-way inter-process communication without deadlock

From communication with Hans Fugal: import std.stdio; import std.range; import std.process; void main() { // exactly how large this needs to be to block the pipeline is probably // somewhat system-dependent, though likely very similar for most // boxen in a given family (linux, osx, etc). string s = repeat('a', 4096 * 64).array; auto p = pipeProcess(["cat"]); p.stdin.write(s); p.stdin.close; writeln(p.stdout.byChunk(4096)); } dtruss -f output (OSX, I can give strace on linux with a little more effort if it makes a difference): ... 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 44424/0x1092df: write_nocancel(0x4, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x4000) = 16384 0 ... 44425/0x1092eb: read(0x0, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x10000) = 65536 0 44425/0x1092eb: write(0x1, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x10000) = 65536 0 44425/0x1092eb: read(0x0, "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", 0x10000) = 65536 0 (hang) Threading and low-level workarounds are of course an option, this isn't a bug. But it would be friendly to take a page from the python playbook and provide something that makes this kind of use case easy. Or at least warn about the possibility and provide a pointer as to possible workarounds in the docs.

How much more buffering do you recommend? Bottom line is there's got to be a limit before blocking enters the picture, otherwise infinite buffering is required, and that's just impossible.

I think a primitive that would use different threads for reading and writing would help.

Hi Brad, the idea is to have something like python's subprocess.Popen.communicate() which (conceptually) writes a string to the subprocess's stdin, and then reads from its stdout and stderr until EOF. This must be implemented with alternating event-driven writes and reads (i.e. using select() - different threads would work but it's unnecessary overhead), to avoid the pipe deadlock that you can get when you try to write everything and then read everything. This is a common pattern that is I think worth making easy. It is not the only pattern, sometimes you need more control and stdin isn't known up-front, you need parallelism and pipelining, etc. But it greatly simplifies the cases where it applies. 1. http://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate

Right. I'm familiar with the problem space. The issue, which python chooses to make it the users problem, and only really visible in docs, if they happen to see it: Note: The data read is buffered in memory, so do not use this method if the data size is large or unlimited. That's a bit more cavalier than I prefer in our standard library. If we require both the input range and the output sink to be supplied, then that puts the choice front and center to the api and not an implementation detail to leave as buyer beware.

Wouldn't a select/threads-based approach take care of the buffering issue?

Andrei, no you have to buffer all of stdout before you return (and passing all of stdin before calling is a form of buffering too). A source/sink approach would work just fine. Perhaps you can already do that with ProcessPipes's stdin/stdout/stderr? I'm still ramping up on the rich stdlib so I'm not sure what constructs but if I had a way to easily plug p.stdin into some kind of source (maybe a string that already exists, i.e. is already buffered), and p.stdout into some kind of sink (maybe just a string that grows (buffers)), and then I have some way to know when it has finished I can extract the stdout. i.e. if it's easy to go string -> process -> string by wiring together a couple existing classes, then all we have to do is document it.

To elaborate, there are 4 buffers involved. Two of them are in the operating system (the pipe buffers), and two are in our space. If we try to flush all of our buffer before reading, then the pipe buffers can fill up and cause deadlock. Brad is saying that just solving the pipe buffers problem doesn't go as far as he'd like - he wants to also solve the problem of having to buffer in the program too, i.e. generally the problem where stdin may be very large (or infinite) and processing stdout doesn't want to have to read all the way to EOF which may be very large (or infinite).

(also I find it somewhat ironic that I am advocating for the equivalent of python's subprocess.Popen.communicate() as 80% of the time I am grumbling that there isn't something more general because communicate() doesn't do what I need, so I am definitely in favor of a more general but still deadlock-free approach)

Restating a little, moving the limits out of the pipe layer into the library layer doesn't solve the underlying fundamental problem, buffering forever is untenable. So, that requires a usage pattern change. Simply increasing the size of the buffers might make more apps work, but it also means that it's more likely that when the still arbitrary limits are hit that it's harder to understand why the limits exist. I prefer low limits here because it makes it more obvious early that the pattern is broken.

How about the following design: void runPipes( InputRange ir1, File if1, InputRange ir2, File if2, ... File of1, OutputRange or1, File of2, OutputRange or2, ... ); This function will take chunks from the input ranges, and asynchronously feed them in the input Files, while at the same time taking any data available to be read from the output Files and sending them to the respective OutputRanges. The function exits when all input ranges are empty, and when the output Files are at eof or closed. On POSIX, this function can use select(). On Windows, I think it's possible to accomplish this using overlapped I/O. Worst case scenario, we can always fall back to threads. Perhaps someone can suggest a better function signature.

Actually that function is also missing File -> File functionality, for inter-process piping. I suppose internally it can abstract away files and ranges.

Ping? Does that design satisfy your requirements? Can I (or someone else) go ahead with implementing it?

No reply to my request for clarification in 3 years, closing.