Bug 20562 – [dmd] Memory allocation failed (ERROR: This is a compiler bug)

After updating to 2.088.1 from 2.087.1, my project no longer compiles with dmd under Windows and just throws an OutOfMemoryError. I bisected it to https://github.com/dlang/dmd/pull/10446, which adds a stack size linker parameter to dmd's compilation, and thus breaks mine. > diff --git a/src/build.d b/src/build.d > index 8b5d59373..592ceb8a8 100755 > --- a/src/build.d > +++ b/src/build.d > @@ -361,13 +361,16 @@ alias dmdExe = memoize!(function(string targetSuffix, string[] extraFlags...) > target = env["DMD_PATH"] ~ targetSuffix; > msg = "(DC) DMD%s %-(%s, %)".format(targetSuffix, dmdSources.map!(e => e.baseName).array); > deps = [versionFile, sysconfDirFile, lexer, backend]; > + string[] platformArgs; > + version (Windows) > + platformArgs = ["-L/STACK:8388608"]; > command = [ > env["HOST_DMD_RUN"], > "-of" ~ target, > "-vtls", > "-J"~env["G"], > "-J../res", > - ].chain(extraFlags).chain(flags["DFLAGS"], sources).array; > + ].chain(extraFlags, platformArgs, flags["DFLAGS"], sources).array; > } > return new DependencyRef(dep); > }); I don't know enough. Assuming it's there for a reason and can't be reverted, can we at least increase the size? Or is there some other way for me to be able to keep using dmd? To test (assuming Windows): > git clone https://github.com/zorael/kameloso > cd kameloso > dub build -c vanilla # lightest build C:\Temp\kameloso>dub build -c vanilla Performing "debug" build using C:\D\dmd2\windows\bin\dmd.exe for x86_64. lu:core 0.1.2: target for configuration "library" is up to date. lu:meld 0.1.2: target for configuration "library" is up to date. dialect 0.3.2: target for configuration "library" is up to date. lu 0.1.2: target for configuration "library" is up to date. kameloso 1.5.0+commit.15.g588f6e55: building configuration "vanilla"... --- ERROR: This is a compiler bug. Please report it via https://issues.dlang.org/enter_bug.cgi with, preferably, a reduced, reproducible example and the information below. DustMite (https://github.com/CyberShadow/DustMite/wiki) can help with the reduction. --- DMD v2.090.0-dirty predefs WithAdminPlugin WithAutomodePlugin WithBashQuotesPlugin WithChanQueriesService WithChatbotPlugin WithConnectService WithCTCPService WithHelpPlugin WithNotesPlugin WithOnelinersPlugin WithPersistenceService WithPipelinePlugin WithPrinterPlugin WithQuotesPlugin WithSedReplacePlugin WithSeenPlugin WithWebtitlesPlugin WithPlugins Have_kameloso Have_dialect Have_lu FlagAsUpdated Have_lu_core Have_lu_meld DigitalMars Windows CRuntime_Microsoft CppRuntime_Microsoft LittleEndian D_Version2 all D_SIMD D_InlineAsm_X86_64 X86_64 Win64 D_LP64 assert D_ModuleInfo D_Exceptions D_TypeInfo D_HardFloat binary C:\D\dmd2\windows\bin\dmd.exe version v2.090.0-dirty config C:\D\dmd2\windows\bin\sc.ini DFLAGS -IC:\D\dmd2\windows\bin\..\..\src\phobos -IC:\D\dmd2\windows\bin\..\..\src\druntime\import -L/OPT:NOICF --- core.exception.OutOfMemoryError@src\core\exception.d(647): Memory allocation failed > dub remove dialect --version=* > dub remove lu --version=* Bisection results: digger: Finding shortest path between 77f556acec92e295ac04e96f800f6d81a8b53ec0 and 2ca17849f4ce2585cb288026241337425385a590... digger: 0 commits (about 0 tests) remaining. digger: 2ca17849f4ce2585cb288026241337425385a590 is the first bad commit commit 2ca17849f4ce2585cb288026241337425385a590 Author: The Dlang Bot <[email protected]> Date: Fri Oct 4 01:17:22 2019 +0200 dmd: Merge pull request #10446 from marler8997/compileDmdBuildD https://github.com/dlang/dmd/pull/10446 Compile dmd exe in build.d merged-on-behalf-of: Nicholas Wilson <[email protected]>

That linker parameter was already present in win32.mak, the PR just moved it to build.d. That bisection seems like a false positive. An OutOfMemoryError usually signals a failure when allocating <dynamic memory> - not stack memory. Did you try to compile your project using -lowmem (which enables the GC for DMD)?

(In reply to moonlightsentinel from comment #1) > That linker parameter was already present in win32.mak, the PR just moved it > to build.d. That bisection seems like a false positive. > > An OutOfMemoryError usually signals a failure when allocating <dynamic > memory> - not stack memory. Did you try to compile your project using > -lowmem (which enables the GC for DMD)? I see. Is there a point in retrying the bisection? -lowmem works and I want to use it, but it doesn't work with dub, which I rely on for dependencies. https://forum.dlang.org/post/[email protected] seems promising but it's still some time before it will be in a downloadable release.

> I see. Is there a point in retrying the bisection? Maybe if you could exactly identify a change that increased the memory consumption *a lot*. Then we could try to improve that specific segement to use less memory. But I would expect it to be a more gradual increase scattered over many small changes. > -lowmem works and I want to use it, but it doesn't work with dub, which I rely on for dependencies Not a dub expert but you could add -lowmem using the dflags configuration in dub.(json|sdl) IIRC.

(In reply to moonlightsentinel from comment #3) > > I see. Is there a point in retrying the bisection? > > Maybe if you could exactly identify a change that increased the memory > consumption *a lot*. Then we could try to improve that specific segement to > use less memory. But I would expect it to be a more gradual increase > scattered over many small changes. > > > -lowmem works and I want to use it, but it doesn't work with dub, which I rely on for dependencies > > Not a dub expert but you could add -lowmem using the dflags configuration in > dub.(json|sdl) IIRC. This was one of the topics of https://forum.dlang.org/post/[email protected]; even if defined in dflags dub doesn't pass -lowmem to dmd. The flag is silently ignored.

Ignore that, it does seem to be doing something now. It still runs out of memory though, will experiment. Apologies for the noise.

This probably doesn't help you, but as it's easy to test: Did you try this with 2.090 as well? There was a regression with the change to build.d that omitted the -O flag on Windows. This has been addressed since.

(In reply to Seb from comment #6) > This probably doesn't help you, but as it's easy to test: > Did you try this with 2.090 as well? There was a regression with the change > to build.d that omitted the -O flag on Windows. This has been addressed > since. I hadn't, so I tested it now with current master (v2.090.0-beta.1-118-g8fc81f40d) and I get the same results. Further I tried a 64-bit compiler (with digger build --model=64), and it runs out of memory too. I've also retried the bisection several times hoping it would land on a different commit, but I get the same one. I even made the tester script test each build three times so as to not miss something due to transient successes. The only thing that works so far is to copy/paste the build command `dub build -v` prints and run it with -lowmem added. Curiously dub with -lowmem DFLAGS does not seem to work. Manually calling dmd with the same command does.

JR how did you do your git bisect? What command were you using to build dmd?

(In reply to Jonathan Marler from comment #8) > JR how did you do your git bisect? What command were you using to build dmd? I used digger to automate the bisection. I might add that I started all this with trying to make a minimal test case using dustmite, but after a while it gets reduced to a vague state that sometimes compiles, sometimes doesn't. The test expression I used to bisect was the build command that `dub build -v` printed, grepped for "Memory allocation failed". I needed to invert the result for digger (a success should be when the string was not detected), so I couldn't just leave it as a normal command and had to make a batch script. I don't know Windows so it may be a roundabout way of doing it, but it works. With my repo cloned into C:\Temp\kameloso, then with a C:\Temp\bisect.bat test script looking like the following: (necessitates having run `dub build` at least once to fetch and build the dependencies) ``` @echo off cd \Temp\kameloso dmd -dip25 -dip1008 -m64 -c -ofk -debug -g -w -d -version=WithAdminPlugin -version=WithAutomodePlugin -version=WithBashQuotesPlugin -version=WithChanQueriesService -version=WithChatbotPlugin -version=WithConnectService -version=WithCTCPService -version=WithHelpPlugin -version=WithNotesPlugin -version=WithOnelinersPlugin -version=WithPersistenceService -version=WithPipelinePlugin -version=WithPrinterPlugin -version=WithQuotesPlugin -version=WithSedReplacePlugin -version=WithSeenPlugin -version=WithWebtitlesPlugin -version=WithPlugins -version=Have_kameloso -version=Have_dialect -version=Have_lu -version=FlagAsUpdated -version=Have_lu_core -version=Have_lu_meld -Isource -IC:\Users\zorael\AppData\Local\dub\packages\dialect-0.3.2\dialect\source -IC:\Users\zorael\AppData\Local\dub\packages\lu-0.1.2\lu\source -IC:\Users\zorael\AppData\Local\dub\packages\lu-0.1.2\lu\source source\kameloso\common.d source\kameloso\constants.d source\kameloso\debugging.d source\kameloso\getopt.d source\kameloso\irccolours.d source\kameloso\kameloso.d source\kameloso\logger.d source\kameloso\main.d source\kameloso\messaging.d source\kameloso\plugins\admin.d source\kameloso\plugins\automode.d source\kameloso\plugins\bashquotes.d source\kameloso\plugins\chanqueries.d source\kameloso\plugins\chatbot.d source\kameloso\plugins\common.d source\kameloso\plugins\connect.d source\kameloso\plugins\ctcp.d source\kameloso\plugins\hello.d source\kameloso\plugins\help.d source\kameloso\plugins\notes.d source\kameloso\plugins\oneliners.d source\kameloso\plugins\package.d source\kameloso\plugins\persistence.d source\kameloso\plugins\pipeline.d source\kameloso\plugins\printer.d source\kameloso\plugins\quotes.d source\kameloso\plugins\sedreplace.d source\kameloso\plugins\seen.d source\kameloso\plugins\twitchbot.d source\kameloso\plugins\webtitles.d source\kameloso\printing.d source\kameloso\terminal.d source\kameloso\thread.d source\kameloso\traits.d -vcolumns 2>&1 | findstr /C:"Memory allocation failed" if %ERRORLEVEL% EQU 0 (exit /b 1) else (exit /b 0) ``` I set up a digger bisect.ini looking like the following: ``` good = [email protected] bad = [email protected] reverse = false tester = C:\Temp\bisect.bat ``` Then I set it off with `dub run digger -- bisect bisect.ini`.

Similar issue: core.exception.OutOfMemoryError@src\core\exception.d(647): Memory allocation failed ---------------- 0x0063C06E in onOutOfMemoryError 0x005A50F3 in Mem at C:\Users\vagrant\clones\dmd\src\dmd\root\rmem.d(134) 0x005A5072 in Mem at C:\Users\vagrant\clones\dmd\src\dmd\root\rmem.d(99) 0x005A3775 in OutBuffer at C:\Users\vagrant\clones\dmd\src\dmd\root\outbuffer.d(82) 0x004F7A0A in void dmd.libmscoff.LibMSCoff.WriteLibToBuffer(dmd.root.outbuffer.OutBuffer*) at C:\Users\vagrant\clones\dmd\src\dmd\libmscoff.d(457) 0x004D8D14 in void dmd.lib.Library.write() at C:\Users\vagrant\clones\dmd\src\dmd\lib.d(106) 0x004FDCF2 in int dmd.mars.tryMain(uint, const(char)**, ref dmd.globals.Param) at C:\Users\vagrant\clones\dmd\src\dmd\mars.d(728) 0x004FE54E in _Dmain at C:\Users\vagrant\clones\dmd\src\dmd\mars.d(944) 0x0063E7EB in void rt.dmain2._d_run_main2(char[][], uint, extern (C) int function(char[][])*).runAll().__lambda1() 0x0063E765 in void rt.dmain2._d_run_main2(char[][], uint, extern (C) int function(char[][])*).runAll()0x0063E600 in _d_run_main2 0x0063B21A in _d_run_main 0x004FE4F5 in main at C:\Users\vagrant\clones\dmd\src\dmd\mars.d(893) 0x0065FCF1 in mainCRTStartup

I'm tempted to close this, as `OutOfMemoryError` cannot be reliably bisected and there's no "action point" here. Don't get me wrong, the compiler IS using too much memory, and it IS absolutely a problem (I run into it every week at least). But it's a structural problem that needs to be addressed, and for the time being the best way to deal with it is to throw `-lowmem` in your dmd.conf. How does that sound ?

(In reply to Mathias LANG from comment #11) > I'm tempted to close this, as `OutOfMemoryError` cannot be reliably bisected > and there's no "action point" here. > > Don't get me wrong, the compiler IS using too much memory, and it IS > absolutely a problem (I run into it every week at least). > > But it's a structural problem that needs to be addressed, and for the time > being the best way to deal with it is to throw `-lowmem` in your dmd.conf. > > How does that sound ? Yes, I can solve it for myself that way, but it becomes somewhat of a barrier of entry to ask users to please change their global settings to be able to compile my project. Is https://issues.dlang.org/show_bug.cgi?id=20699 very difficult to solve? I already set up build configurations like `application-lowmem` for the purpose, they just effectively don't do anything yet. Can a dmd.conf in the working directory append to DFLAGS from the system-wide /etc/dmd.conf? Instead of overriding? > DFLAGS+="-lowmem"

I may have come across as unappreciative, which was not my intention. (In reply to Mathias LANG from comment #11) > But it's a structural problem that needs to be addressed, and for the time > being the best way to deal with it is to throw `-lowmem` in your dmd.conf. > > How does that sound ? Right now I am unable to reproduce the error (2.095), but I only did a casual test. Regardless, please close this if the issue is known and having an open bug report about it only amounts to noise.