Bug 150 – (D1 only) std.gc.minimize doesn't minimize physical memory usage

As shown by the code below, std.gc.minimize doesn't have reduce the physical memory usage as advertised. import std.gc; import std.stdio; void printGC(){ GCStats stat; getStats(stat); writefln("poolsize: %s; usedsize: %s; freeblocks: %s; freelistsize: %s; pageblocks: %s\n", stat.poolsize, stat.usedsize, stat.freeblocks, stat.freelistsize, stat.pageblocks); } int main(){ printGC(); size_t len = 256 * 1024 * 1024; byte* b = new byte[len]; writefln("after allocating %s bytes", len); printGC(); delete b; b = null; fullCollect(); writefln("after std.gc.fullCollect:"); printGC(); minimize(); writefln("after std.gc.minimize:"); printGC(); return 0; }

updated test code to DMD-1.007 # import gcstats; # import std.gc; # import std.stdio; # # void printGC(){ # GCStats stat; # # getStats(stat); # # writefln("poolsize: %s; usedsize: %s; freeblocks: %s; " # "freelistsize: %s; pageblocks: %s\n", # stat.poolsize, stat.usedsize, stat.freeblocks, # stat.freelistsize, stat.pageblocks); # } # # int main(){ # printGC(); # # size_t len = 256 * 1024 * 1024; # byte* b = (new byte[len]).ptr; # # writefln("after allocating %s bytes", len); # printGC(); # # delete b; # b = null; # # fullCollect(); # writefln("after std.gc.fullCollect:"); # printGC(); # # minimize(); # writefln("after std.gc.minimize:"); # printGC(); # # return 0; # }

This is fixed in D2 and here is a patch based on druntime for Phobos 1: --- gcx.orig.d 2010-08-15 21:42:21.837000089 -0300 +++ gcx.new.d 2010-08-15 21:52:24.319000075 -0300 @@ -1038,7 +1038,10 @@ class GC void minimize() // minimize physical memory usage { - // Not implemented, ignore + synchronized (gcLock) + { + gcx.minimize(); + } } void setFinalizer(void *p, GC_FINALIZER pFn) @@ -2252,6 +2255,39 @@ struct Gcx } + /** + * Minimizes physical memory usage by returning free pools to the OS. + */ + void minimize() + { + size_t n; + size_t pn; + Pool* pool; + size_t ncommitted; + + for (n = 0; n < npools; n++) + { + pool = pooltable[n]; + ncommitted = pool.ncommitted; + for (pn = 0; pn < ncommitted; pn++) + { + if (cast(Bins)pool.pagetable[pn] != B_FREE) + break; + } + if (pn < ncommitted) + continue; + pool.Dtor(); + cstdlib.free(pool); + memmove(pooltable + n, + pooltable + n + 1, + (--npools - n) * (Pool*).sizeof); + n--; + } + minAddr = pooltable[0].baseAddr; + maxAddr = pooltable[npools - 1].topAddr; + } + + /** * Run finalizer on p when it is free'd. */ It might be probably a good idea to call minimize() after fullcollect() too.

Assigning this to Sean.

import core.memory; import std.stdio; struct GCStats { size_t poolsize; // total size of pool size_t usedsize; // bytes allocated size_t freeblocks; // number of blocks marked FREE size_t freelistsize; // total of memory on free lists size_t pageblocks; // number of blocks marked PAGE } extern(C) GCStats gc_stats(); void printGC(){ auto stat = gc_stats(); writefln("poolsize: %s; usedsize: %s; freeblocks: %s; freelistsize: %s; pageblocks: %s\n", stat.poolsize, stat.usedsize, stat.freeblocks, stat.freelistsize, stat.pageblocks); } int main(){ printGC(); size_t len = 256 * 1024 * 1024; auto b = new byte[len]; writefln("after allocating %s bytes", len); printGC(); delete b; b = null; GC.collect(); writefln("after std.gc.fullCollect:"); printGC(); GC.minimize(); writefln("after std.gc.minimize:"); printGC(); return 0; } ---------- poolsize: 65536; usedsize: 256; freeblocks: 15; freelistsize: 3840; pageblocks: 0 after allocating 268435456 bytes poolsize: 268566528; usedsize: 256; freeblocks: 30; freelistsize: 3840; pageblocks: 1 after std.gc.fullCollect: poolsize: 268566528; usedsize: 256; freeblocks: 65567; freelistsize: 3840; pageblocks: 0 after std.gc.minimize: poolsize: 65536; usedsize: 256; freeblocks: 15; freelistsize: 3840; pageblocks: 0 ---------- Still fixed in D2 and D1 also has an implementation https://github.com/D-Programming-Language/druntime/blob/D1.0/src/gc/basic/gcx.d#L1917.

Recoding the test case for D1 Phobos: ------------------- import std.gc; import std.stdio; import gcstats; extern(C) GCStats gc_stats(); void printGC(){ GCStats stat; getStats(stat); writefln("poolsize: %s; usedsize: %s; freeblocks: %s; freelistsize: %s; pageblocks: %s\n", stat.poolsize, stat.usedsize, stat.freeblocks, stat.freelistsize, stat.pageblocks); } int main(){ printGC(); size_t len = 256 * 1024 * 1024; auto b = new byte[len]; writefln("after allocating %s bytes", len); printGC(); delete b; b = null; std.gc.fullCollect(); writefln("after std.gc.fullCollect:"); printGC(); std.gc.minimize(); writefln("after std.gc.minimize:"); printGC(); return 0; }

I folded in Leandro's patch, but the problem remains. (D1 only.) https://github.com/D-Programming-Language/phobos/commit/25616cf9790084a7b1b3db6d3aa8e910b9195445

(In reply to comment #6) > I folded in Leandro's patch, but the problem remains. > > (D1 only.) > > https://github.com/D-Programming-Language/phobos/commit/25616cf9790084a7b1b3db6d3aa8e910b9195445 So, that druntime branch is not really used in D1, right? Maybe you should remove it to avoid confusion. https://github.com/D-Programming-Language/druntime/blob/D1.0/src/gc/basic/gcx.d#L1917.

BTW, seems to be working: --- ~/dmd/dmd-1.072/dmd/src/phobos:phobos-1.x %=$ ./dmd | head -n1 DMD64 D Compiler v1.072 ~/dmd/dmd-1.072/dmd/src/phobos:phobos-1.x %=$ cat test.d import std.gc; import std.stdio; import gcstats; extern(C) GCStats gc_stats(); void printGC(){ GCStats stat; getStats(stat); writefln("poolsize: %s; usedsize: %s; freeblocks: %s; freelistsize: %s; pageblocks: %s\n", stat.poolsize, stat.usedsize, stat.freeblocks, stat.freelistsize, stat.pageblocks); } int main(){ printGC(); size_t len = 256 * 1024 * 1024; auto b = new byte[len]; writefln("after allocating %s bytes", len); printGC(); delete b; b = null; std.gc.fullCollect(); writefln("after std.gc.fullCollect:"); printGC(); std.gc.minimize(); writefln("after std.gc.minimize:"); printGC(); return 0; } ~/dmd/dmd-1.072/dmd/src/phobos:phobos-1.x %=$ ./dmd -run test poolsize: 65536; usedsize: 512; freeblocks: 14; freelistsize: 7680; pageblocks: 0 after allocating 268435456 bytes poolsize: 268566528; usedsize: 512; freeblocks: 29; freelistsize: 7680; pageblocks: 1 after std.gc.fullCollect: poolsize: 268566528; usedsize: 512; freeblocks: 65566; freelistsize: 7680; pageblocks: 0 after std.gc.minimize: poolsize: 65536; usedsize: 512; freeblocks: 14; freelistsize: 7680; pageblocks: 0 --- If you want to run a minimize() after a collection, apply this pull request: https://github.com/D-Programming-Language/phobos/pull/397 After the patch: --- poolsize: 65536; usedsize: 512; freeblocks: 14; freelistsize: 7680; pageblocks: 0 after allocating 268435456 bytes poolsize: 268566528; usedsize: 512; freeblocks: 29; freelistsize: 7680; pageblocks: 1 after std.gc.fullCollect: poolsize: 65536; usedsize: 512; freeblocks: 14; freelistsize: 7680; pageblocks: 0 after std.gc.minimize: poolsize: 65536; usedsize: 512; freeblocks: 14; freelistsize: 7680; pageblocks: 0 ---

And BTW, I'm not dawg :)

Copying comment by Walter on the pull request: > It's still not working, at least on Windows. What does that means? What output are you getting. (I changed the issue as Windows only)

poolsize: 65536; usedsize: 192; freeblocks: 15; freelistsize: 3904; pageblocks: 0 after allocating 268435456 bytes poolsize: 268566528; usedsize: 192; freeblocks: 30; freelistsize: 3904; pageblocks: 1 after std.gc.fullCollect: poolsize: 268566528; usedsize: 192; freeblocks: 65567; freelistsize: 3904; pageblocks: 0 after std.gc.minimize: poolsize: 268566528; usedsize: 192; freeblocks: 65567; freelistsize: 3904; pageblocks: 0

It's unlikely this D1 issue will get worked on, if anyone plans to work on it feel free to reopen.