Bug 19322 – A lot of memory is consumed and not freed to the system when Exception is formatted with stacktrace in debug

Original issue filed in vibe-d: https://github.com/vibe-d/vibe.d/issues/2221 Test code: ``` #!/usr/bin/env dub /+ dub.sdl: name "memleak" dependency "vibe-d:core" version="~>0.8.4" subConfiguration "vibe-d:core" "libevent" +/ import core.memory; import std.stdio; import std.experimental.logger; import vibe.core.core; // comment this out to behave normally void main() { foreach (i; 0..1000) { try throw new Exception("some message"); catch (Exception ex) { errorf("Error: %s", ex); //errorf("Error: %s", ex.msg); } } info("Done"); GC.collect(); GC.minimize(); readln(); } ``` Can be run with: dub --single sample.d If I run this with dmd >= 2.081.0 the process is almost 500MB in memory. GC.collect() and GC.minimize() doesn't matter. If run with new vibe-core it's somewhat better at about 400MB in memory. When just exception message is logged, process is at 6MB - what a difference. Same when run with dmd-2.080.1 or without vibe-d. Tested on fedora 28 x86_64 with multiple dmd versions and vibe-d-0.8.4. Tested also on Ubuntu 18.04.1 x86_64 I tried to bisect this using digger with: bad = https://github.com/dlang/phobos/pull/6700 good = https://github.com/dlang/phobos/pull/4941 And the winner seems to be: https://github.com/dlang/druntime/pull/2169 And due to it's change context it seems to be really connected with this even though it should affect only OSX. dmd --profile=gc and valgrind tools doesn't seems to lead anywhere. What I've found after all is that it is unrelated to vibe-d (so I moved it here). If tried with for example pretty large botan library: dependency "botan" version="~>1.12.10" Resulted process memory (VmRSS) is much larger. So it seems to be some problem with size of the included libraries and probably some memory fragmentation when building the stacktrace? It's ok in release build.

This is probably because at least for ELF, the .debug_line section data is mmap()ped from the executable file and not unmapped after looking up file and line infos for the frames in the backtrace. The mmapped section data is owned by the ElfSection struct (via its its MMapRegion member) created in the return statement in Image.getDebugLineSectionData(). What's actually returned is the data array via alias this, and the ElfSection struct (and its MMapRegion member) apparently aren't destructed, which seems to be a bug in its own right. I only know because I 'had' to replicate this leaking behavior in https://github.com/dlang/druntime/pull/2330/files#diff-a10edf024597f176f219ea3c0cb2bdafR64, otherwise the data array was invalid (unmapped) as soon as the data slice was returned. Possible fix: pass a delegate processing the section data (something like `Image.processDebugLineSectionData(dg)`), so that the Image type can hold and release the data itself.

Thanks for the lead description. It seems to be exactly it. I've tried to debug this a bit to understand this more. And the problem is as you said with this line: https://github.com/dlang/druntime/blob/master/src/rt/backtrace/elf.d#L59 which thanks to the alias here: https://github.com/dlang/druntime/blob/master/src/rt/backtrace/elf.d#L197 returns only aliased ubyte[] here: https://github.com/dlang/druntime/blob/master/src/rt/backtrace/elf.d#L59 and that doesn't destruct MMaped file which is hold here: https://github.com/dlang/druntime/blob/master/src/rt/backtrace/elf.d#L199 which seems to be another bug (I've tried to reproduce it in a simple test case, but failed to do so). But even if it will be destroyed on return, than the debugLineSectionData would point to invalid data.

I already fixed this as part of a mainly refactoring, still pending PR: https://github.com/dlang/druntime/pull/2330

@kinke updated dlang/druntime pull request #2330 "Refactor ELF utils from rt.backtrace.elf and rt.sections_elf_shared to core.elf" fixing this issue: - Fix Issue 19322 (memory leak in backtrace generation with -g) https://github.com/dlang/druntime/pull/2330

dlang/druntime pull request #2330 "Refactor ELF utils from rt.backtrace.elf and rt.sections_elf_shared to core.internal.elf.{dl,io}" was merged into master: - 0fa014f03cdfd24e252ee8c0b49a5d9fe0444d8f by Martin Kinkelin: Fix Issue 19322 (memory leak in backtrace generation with -g) https://github.com/dlang/druntime/pull/2330