Bug 18005 – AA leak

import std.stdio, std.typecons, std.array, std.conv, std.algorithm, std.range; void main(string[] args) { uint[string] vv; foreach(a; 20_000.iota.map!(a => a * 50_000)) { vv[a.to!string] = a; } uint cnt; while(true) { import core.memory; writefln(`%g MB used, %g MB free`, GC.stats.usedSize / 1024f / 1024, GC.stats.freeSize / 1024f / 1024); new ubyte[13 * 1024 * 1024]; GC.collect; GC.minimize; if(cnt++ == 30) break; } } On windows: 0.860535 MB used, 4.13947 MB free 13.8644 MB used, 10.6395 MB free 26.8683 MB used, 17.1395 MB free 39.8723 MB used, 23.6395 MB free 52.8762 MB used, 30.1395 MB free 65.8801 MB used, 36.6395 MB free 78.884 MB used, 45.6356 MB free 91.8879 MB used, 57.6317 MB free 104.892 MB used, 72.6277 MB free 117.896 MB used, 59.6238 MB free 130.9 MB used, 77.6199 MB free 143.904 MB used, 64.616 MB free 156.907 MB used, 85.6121 MB free 169.911 MB used, 72.6082 MB free 182.915 MB used, 96.6043 MB free 195.919 MB used, 83.6004 MB free 208.923 MB used, 110.596 MB free 221.927 MB used, 97.5926 MB free 234.931 MB used, 84.5887 MB free 247.935 MB used, 114.585 MB free 260.939 MB used, 101.581 MB free 273.943 MB used, 88.577 MB free 286.946 MB used, 121.573 MB free 299.95 MB used, 108.569 MB free 312.954 MB used, 95.5652 MB free 325.958 MB used, 131.561 MB free 338.962 MB used, 118.557 MB free 351.966 MB used, 105.554 MB free 364.97 MB used, 144.55 MB free 377.974 MB used, 131.546 MB free 390.978 MB used, 118.542 MB free And on linux: 1.41583 MB used, 3.58417 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free 14.4197 MB used, 10.0842 MB free

Are you using -m64 on windows? 32-bit OSes are well known for having memory leaks due to false pointers.

Yes. Also there's NO_SCAN attribute so there should not be any.

There's nothing different on Windows vs. Linux with regards to the AA implementation. Both are abstracted against the GC. So if this is 64-bit windows, the only attribute I can think of to associate with this behavior is the numbers the OS assigns to stacks/heaps happen to coincide with the pointers that are there. (In reply to Temtaime from comment #2) > Also there's NO_SCAN attribute so there should not be any. Doesn't matter. The stack is not NO_SCAN, and the AA bucket elements are not NO_SCAN. What you are doing is making large targets (13MB targets) that these false pointers may hook onto. The nature of false pointers is that in certain situations, they happen, and changing seemingly random things can affect it. Not sure if we will get to the bottom of this, or even solve it.

don't think is the AA that does it, as the following exposes the same behavior: +++++++++++++++++++++++++++++++++++++++ void main(string[] args) { foreach (i; 0..300) { import core.memory; import std.stdio : writefln; writefln(`%g MB used, %g MB free`, GC.stats.usedSize / 1024f / 1024, GC.stats.freeSize / 1024f / 1024); new bool[13 * 1024 * 1024]; // <- this collects differently on Windows GC.collect; GC.minimize; } } +++++++++++++++++++++++++++++++++++++++ Running it: 0.000137329 MB used, 0.999863 MB free 13.004 MB used, 7.49988 MB free 26.0079 MB used, 13.9999 MB free 39.0118 MB used, 20.4999 MB free 52.0157 MB used, 26.9999 MB free 65.0197 MB used, 33.4999 MB free 78.0236 MB used, 39.9999 MB free 78.0236 MB used, 42.496 MB free 91.0275 MB used, 51.4921 MB free 104.031 MB used, 63.4882 MB free 104.031 MB used, 66.4882 MB free 117.035 MB used, 53.4843 MB free 130.039 MB used, 68.4803 MB free 143.043 MB used, 55.4764 MB free 143.043 MB used, 86.4764 MB free 143.043 MB used, 55.4764 MB free 143.043 MB used, 86.4764 MB free 143.043 MB used, 55.4764 MB free 143.043 MB used, 86.4764 MB free 143.043 MB used, 55.4764 MB free 143.043 MB used, 86.4764 MB free 143.043 MB used, 55.4764 MB free 143.043 MB used, 86.4764 MB free 143.043 MB used, 55.4764 MB free 143.043 MB used, 86.4764 MB free ................................ It stabilizes at 143.043 MB On Linux it collects eagerly and stabilizes around 13.0042 MB used, 7.49976 MB free

The behavior varies across platforms because it very much depends on the actual addresses returned by the OS for allocating memory: linux prefers addresses at the top of the possible addressrange (2^^47) while windows uses returns low addresses above a couple GB (but not reproducable due to randomization). OSX is even worse as it starts with very addresses. In the reporte example, a wide range of low addresses are written into the (key,value) pairs of an AA. As the key is a string, it contains indirections that need to be scanned, i.e. the full allocation is scanned by a conservative GC. Precise GC to the rescue: when calling the build with "--DRT-gcopt=gc:precise --DRT-scanDataSeg=precise" as of dmd2.085, even for win32 it behaves as the linux build. I guess this is the best we can do, so closing...