Bug 10906 – [2.064 git-head] Out of memory compiling Phobos on Windows

Using the latest github masters of DMD, druntime and phobos, compiling Phobos on my Windows machine with 4GB RAM fails with "Error: out of memory". Despite that, compiling the same versions inside a Linux (Debian 6) VirtualBox VM *inside the same Windows machine* works fine.

Did you figure out what was causing the OOM? I'll bet it's either std.datetime or std.algorithm (likely the latter if you're not running unittests -- std.datetime is very unittest-heavy, but it's std.algorithm that has exponential template expansion in some places *cough* namely cartesianProduct, written by yours truly).

The OOM is occurring during this (rather large) command: --------------------------------- dmd -lib -ofphobos.lib -Xfphobos.json -O -release -w -d std\stdio.d std\stdiobase.d std\string.d std\format.d std\file.d std\range.d std\array.d std\functional.d std\path.d std\outbuffer.d std\utf.d std\csv.d std\math.d std\complex.d std\numeric.d std\bigint.d std\metastrings.d std\bitmanip.d std\typecons.d std\uni.d std\base64.d std\md5.d std\ascii.d std\demangle.d std\uri.d std\mmfile.d std\getopt.d std\signals.d std\typetuple.d std\traits.d std\encoding.d std\xml.d std\random.d std\exception.d std\compiler.d std\system.d std\concurrency.d std\datetime.d std\uuid.d std\digest\crc.d std\digest\sha.d std\digest\md.d std\digest\ripemd.d std\digest\digest.d std\algorithm.d std\variant.d std\syserror.d std\zlib.d std\stream.d std\socket.d std\socketstream.d std\container.d std\conv.d std\zip.d std\cstream.d std\regex.d std\stdint.d std\json.d std\parallelism.d std\mathspecial.d std\process.d crc32.d std\net\isemail.d std\net\curl.d std\c\process.d std\c\stdlib.d std\c\time.d std\c\stdio.d std\c\math.d std\c\stdarg.d std\c\stddef.d std\c\fenv.d std\c\string.d std\c\locale.d std\c\wcharh.d std\windows\registry.d std\windows\iunknown.d std\windows\syserror.d std\windows\charset.d std\c\windows\windows.d std\c\windows\com.d std\c\windows\winsock.d std\c\windows\stat.d std\internal\processinit.d std\internal\uni.d std\internal\uni_tab.d std\internal\unicode_tables.d std\internal\digest\sha_SSSE3.d std\internal\math\biguintcore.d std\internal\math\biguintnoasm.d std\internal\math\biguintx86.d std\internal\math\gammafunction.d std\internal\math\errorfunction.d std\internal\windows\advapi32.d etc\c\zlib.d etc\c\curl.d etc\c\sqlite3.d etc\c\zlib\zlib.lib ..\druntime\lib\druntime.lib --------------------------------- Beyond that, I really don't know. Maybe the posix makefile splits it into smaller chunks than that? Maybe a compiler change resulted in greater memory usage on Windows? Maybe posix just handles low memory better? I'm not sure I'd even know where to start.

No, in Linux it also runs that huge compile line. One way to test is perhaps to copy-n-paste that command, omitting likely culprits like std.algorithm and std.datetime (or std.regex, due to ctRegex, but it's less likely) to see if it makes a difference.

Oh, and add -c to the command line as well, to prevent it from erroring out from undefined symbols.

If the compiler attempts to use more than 2GB of virtual memory (including all loaded libraries), then windows will throw an OOM error for exe's which aren't marked as Large Address Aware. (such as dmd) As far as I know, posix systems don't have this limitation and can use the full 4gb of the 32-bit address space without needing to set a flag on the executable. I've managed to trigger this error compiling one of my larger libraries (the arguments actually have to be written to file before they can be passed to dmd, otherwise the command line is too long) it ended up taking roughly 2.5gb of RAM (after I set the Large Address Aware flag) for 2.63.2 to compile my 3.8mb across 1.3k heavily interdependent modules into a static library. Perhaps the Large Address Aware flag should be set by default on dmd?

Another thing to try, based on a wild gueses, is to edit std/algorithm.d and add version(none) to cartesianProduct's unittests. One of them, I believe, has a triple cartesian product, which in the current implementation causes an explosion in the number of templates that must be instantiated.

Running the original command but with both std.algorthm and std.datetime removed does succeed. Also, running the same command, but with *only* std.algorthm and std.datetime *also* succeeds. So it would seem to be the combination of everything that pushes it over. However, it still seems a little strange, though: When I run the original command, with everything, the memory usage only goes up to just over 600MB before failing with "out of memory". And I have a couple gigs free, plus virtual memory. Additionally, it works fine on Linux, on the same physical machine, running *inside* a VM (set up with only 768MB RAM system memory). (FWIW, I *am* still getting the OOM on Windows even when the Linux VM isn't running - so it's definitely not a case of the VM using up too much of the available RAM.)

How do I set the Large Address Aware flag? I'm having trouble finding that info through a web search.

Actually, if it works with std.algo and atd.datetime done separately (even without the large address aware flag), and if those are notorious for being large, perhaps I should just toss together a pull request adjusting phobos's makefiles to do those separately? Or are there known issues with that?

Like this, maybe: http://www.pso-world.com/forums/showthread.php?t=199993 It is a single bit in the exe header, you could do it with a hex editor if you really wanted to.

(In reply to comment #9) > Actually, if it works with std.algo and atd.datetime done separately (even > without the large address aware flag), and if those are notorious for being > large, perhaps I should just toss together a pull request adjusting phobos's > makefiles to do those separately? Or are there known issues with that? You should toss together a pull request that splits std.algorithm into std.algorithm.package, std.algorithm.search, std.algorithm.sort, std.algorithm.set, and std.algorithm.mutation. :-) Seriously, it's time we cut up that monster into more manageable pieces, now that we have package.d. I would say the same for std.datetime as well, but I believe Jonathan is already working on that one.

> I would say the same for std.datetime as well, but I believe Jonathan is > already working on that one. I have it most of the way done already, but I'm quite busy at the moment, so finishing it is going slowing.

(In reply to comment #11) > (In reply to comment #9) > > Actually, if it works with std.algo and atd.datetime done separately (even > > without the large address aware flag), and if those are notorious for being > > large, perhaps I should just toss together a pull request adjusting phobos's > > makefiles to do those separately? Or are there known issues with that? > > You should toss together a pull request that splits std.algorithm into > std.algorithm.package, std.algorithm.search, std.algorithm.sort, > std.algorithm.set, and std.algorithm.mutation. > That wouldn't address this out-of-memory issue though, would it? I started modifying the makefiles to break up the compilation into smaller steps, but I don't know how to do it without breaking the output of phobos.json. We don't have any way to build phobos.json progressively, do we?

I'm closing this because the immediate issue is fixed by: https://github.com/D-Programming-Language/dmd/pull/2507 However, that may not be a permanent solution: As phobos continues to grow, the problem may resurface unless we address the issues of DMD memory usage and adjusting the makefile to not build all of Phobos at once.