Have a 1G file named "one_gigabyte_file". You can use the following program to create it:
import std.stdio;
void foo()
{
auto file = File("one_gigabyte_file", "w");
auto data = new ubyte[](100 * 1024 * 1024);
foreach (i; 0 .. 10) {
file.rawWrite(data);
}
}
void main()
{
foo();
}
Then, run the following program:
import std.stdio;
void foo()
{
auto file = File("one_gigabyte_file", "r");
auto data = new ubyte[](100 * 1024 * 1024);
foreach (i; 0 .. 10) {
file.rawRead(data);
}
}
void main()
{
for (size_t i; true; ++i) {
writeln(i);
foo();
}
}
The problem is, the memory consumption of the program grows continuously until it is terminated by core.exception.OutOfMemoryError.
We don't see this issue under Linux.
Thank you,
Ali
Comment #1 by bugzilla — 2012-11-28T18:05:14Z
The garbage collector is completely unsuited for managing gigantic allocations like these. The reason is that a 1 gig allocation consumes 25% of the 32 bit address space.
Since the GC is a conservative mark-sweep one, 25% of any random bit patterns are going to be assumed to be a reference into those allocations, preventing them from being collected.
The only way out of this is with precise GC for everything, including the stack and static data segments. This will be impractical to achieve, if for no other reason than making it hard to interoperate with C code when sharing pointers between C and D.
Such large allocations need to be manually managed, with something like malloc/free. I don't see this changing in the near future.
Comment #2 by doob — 2012-11-28T23:28:33Z
(In reply to comment #1)
> The only way out of this is with precise GC for everything, including the stack
> and static data segments. This will be impractical to achieve, if for no other
> reason than making it hard to interoperate with C code when sharing pointers
> between C and D.
Isn't it possible to do some kind of special treatment when integrating with C. I'm talking about the developer need to do something special, i.e. call GC.addRoot or similar. I don't think we should limit pure D programs, if possible.
Comment #3 by bugzilla — 2012-11-29T02:17:27Z
Yes, it's possible, but D has set a store by easy interoperability with C, and my experience is that having a protocol like that is doomed to misuse and bugs.
Comment #4 by Jesse.K.Phillips+D — 2016-11-03T20:24:38Z
I'm having this issue with Windows 64bit on 30MB files, and it appears is also affecting zip extraction of similar size. I'm reopening so the issue can be re-evaluated considering this isn't just a 32bit + gigabyte file issue.
Does anyone have a suggestion for manually releasing the expanded data when using std.zip.ZipArchive.expand() on a ArchiveMember?
Comment #5 by r.sagitario — 2019-04-21T13:31:57Z
With the precise GC enabled via "--DRT-gcopt=gc:precise --DRT-scanDataSeg=precise"
the program no longer runs out of memory, but needs up to 500MB of memory (the 64-bit version runs with less than 200MB). Probably still some false pointers on the stack.
Comment #6 by robert.schadek — 2024-12-07T13:32:11Z