Bug 1980 – memory allocated via double[][] is not being freed by the GC

Hi, For a data processing application I need to read a large number of data sets from disk. Due to their size, they have to be read and processed sequentially, i.e. in pseudocode int main() { while (some condition) { double[][] myLargeDataset = read_data(); process_data(myLargeDataset); // free all memory here otherwise next cycle will // run out of memory } return 0; } Now, the "problem" is the fact that each single data-set saturates system memory and hence I need to make sure that all memory is freed after each process_data step is complete. Unfortunately, using dmd-2.012 I have not been able to achieve this. Whatever I do (including nothing, i.e., letting the GC do its job), the resulting binary keeps accumulating memory and crashing shortly after. I've tried deleting the array, setting it to null, setting the array lengths to 0, and manually forcing the GC to collect, to no avail. Hence, is there something I am doing terribly wrong or is this a bug in dmd? Thanks much, Markus

Below is a complete piece of code that still exhibits this issue. The file "data_random.dat" simply contains a single row of random ints (about 30M filesize). Each iteration allocated additional heap and I don't seem to be able to free foo no matter what. import std.stream; import std.stdio; import std.contracts; import std.gc; double[][] parse(BufferedFile inputFile) { double[][] array; foreach(char[] line; inputFile) { double[] temp; foreach(string item; std.string.split(assumeUnique(line))) { temp ~= std.string.atof(item); } array ~= temp; } /* rewind for next round */ inputFile.seekSet(0); return array; } int main() { BufferedFile inputFile = new BufferedFile("data_random.dat"); while(1) { double[][] foo = parse(inputFile); } return 1; }

The test program works fine for me on Windows. I tried it with DMD/Phobos 1.028, DMD/Tango/Tangobos 1.028, and DMD/Phobos 2.012. Although the data file I tested it with was only 50K. That could make a difference if it's being caused by "false pointers". Or it could be a Linux-only bug, I suppose. Maybe for the lazy you could also modify your program to generate a 30MB file full of random numbers. :-)

Thanks much for testing Bill! I just ran both the 32bit DMD executable and the 64bit GDC (v. 0.24) version of the above code on my opteron box. The 32bit DMD version leaks memory whereas the GDC binary works just fine and properly free its memory. Thanks, Markus

It's possible that this might be related to issue 1804 (http://d.puremagic.com/issues/show_bug.cgi?id=1804)

Either a duplicate of issue #1804 or issue #3463. *** This issue has been marked as a duplicate of issue 1804 ***