Bug 23603 – ICE out of memory when using -lowmem

The code below makes dmd throw an OutOfMemoryError when using the -lowmem option. ------------------------main.d------------------- struct Array(T) { T[] _payload; this(int b) { import std; _payload.length = b; } } struct A { ushort[131070] a; } auto data = Array!A(255); -------------------------------------------------- Invocation: c:\d\dmd.2.101.2.windows\dmd2\windows\bin\dmd.exe -lowmem -c main.d Memory usage climbs to ca. 3.5 GB with lots of available RAM to spare before giving up with: --- ERROR: This is a compiler bug. Please report it via https://issues.dlang.org/enter_bug.cgi with, preferably, a reduced, reproducible example and the information below. DustMite (https://github.com/CyberShadow/DustMite/wiki) can help with the reduction. --- DMD v2.101.2-dirty predefs DigitalMars LittleEndian D_Version2 all Windows Win32 CRuntime_Microsoft CppRuntime_Microsoft D_InlineAsm D_InlineAsm_X86 X86 assert D_PreConditions D_PostConditions D_Invariants D_ModuleInfo D_Exceptions D_TypeInfo D_HardFloat binary c:\d\dmd.2.101.2.windows\dmd2\windows\bin\dmd.exe version v2.101.2-dirty config c:\d\dmd.2.101.2.windows\dmd2\windows\bin\sc.ini DFLAGS -Ic:\d\dmd.2.101.2.windows\dmd2\windows\bin\..\..\src\phobos -Ic:\d\dmd.2.101.2.windows\dmd2\windows\bin\..\..\src\druntime\import --- core.exception.OutOfMemoryError@core\lifetime.d(126): Memory allocation failed ---------------- Without the -lowmem option an equal amount of memory is consumed before succeeding. We are however dependent on using -lowmem due to the size of the code base that this test was reduced from.

Issues: 1. `import std` imports all of Phobos. This is quite impractical, and import by package should never have been added. It will always make compiling desperately slow and will consume vast quantities of memory. Recommendation: just import modules that are needed 2. The declaration of `data` has an initialization that must happen at compile time. The CTFE to do it does not store expressions efficiently - the array is stored as (256 * 131,070) Expression AST nodes. This is going to consume a lot of memory, and will also be extremely slow. Then, it will have to generate (256 * 131,070) static initializers. Recommendation: Replace with a pointer initialized at runtime: Array!A(255)* pdata; static this() { pdata = new Array!(255); } which will push that all off to runtime, where it is efficiently handled. This should resolve the slow compiles, vast compile time memory consumption, and large executable file size problems. I don't see any reasonable way to fix this in the language.

I agree with your recommendations obviously, but it doesn't solve the seemingly unexplainable cases where dmd claims to run out of memory. There is plenty of memory available. This example succeeds when I *remove* the -lowmem option. LDC does not run out of memory compiling the same code. In other cases that are hard to reduce I sometimes get the above verbose message, sometimes just "Error: out of memory", sometimes only exit code -1073741795 (0xC000001D). When a large code base reduces to a nice concise handfull of lines like this it is easy to rewrite like you suggest, but this particular reduction took four weeks to prepare and required manual steps. I have another case that is still running, not sure if that will ever converge to reveal the problematic code.

Thanks to @kinke for pointing this out, I have been reducing with the wrong compiler. Adjusting the path to c:\d\dmd.2.101.2.windows\dmd2\windows\bin64\dmd.exe using the 64bit compiler this compiles fine. (In my defence I started with a legitimate problem, I changed to the latest compiler version halfway the reduction, but I did it wrong.) This particular issue is invalid.