Bug 11021 – [GC] Huge GC leak leads to crash; GC uses 7x more memory

import core.memory; import std.traits; void main() { while(true) { GC.collect(); new ubyte[40 * 1024 * 1024]; } } Application's memory usage grows and it crashes with message: core.exception.OutOfMemoryError If i comment import std.traits; everything goes OK, but now application uses 300 MM of memory.

Are you, by any chance, on win32? What is your metric for "GC uses 7x more memory"? Also, you say: "but now application uses 300 MM of memory." But is that a problem? I don't know what to make about your "import" issue, as I can't reproduce on linux. I'll try on windows.

(In reply to comment #1) > Are you, by any chance, on win32? I can reproduce on win32, all version of dmd. > What is your metric for "GC uses 7x more memory"? > > Also, you say: "but now application uses 300 MM of memory." But is that a > problem? Never mind, I understood what you meant. > I don't know what to make about your "import" issue, as I can't reproduce on > linux. I'll try on windows. I don't know how you did it, but I run out of memory regardless.

Yes, it's on win32. I think app should use about 50 mb as i'm allocating 40 mb array, not 300.

(In reply to comment #3) > Yes, it's on win32. > > I think app should use about 50 mb as i'm allocating 40 mb array, not 300. I would actually expect it to use 90mb, your allocating a 40mb array inside a loop, the previous 40mb array will still be referenced when you go to allocate the new 40mb array, meaning that it can't be collected if the second allocation triggers a GC.

In loop i calling collect, so gc should frees the memory.

(In reply to comment #5) > In loop i calling collect, so gc should frees the memory. You should really be clearing the existing array rather than allocating a new one, but that's beside the point. When you call the GC manually, the array is still referenced in local variable, if you were to reset that local variable before calling the GC, then it should collect it correctly.

(In reply to comment #4) > I would actually expect it to use 90mb, your allocating a 40mb array inside a > loop, the previous 40mb array will still be referenced when you go to allocate > the new 40mb array, meaning that it can't be collected if the second allocation > triggers a GC. (In reply to comment #6) > You should really be clearing the existing array rather than allocating a new > one, but that's beside the point. When you call the GC manually, the array is > still referenced in local variable, if you were to reset that local variable > before calling the GC, then it should collect it correctly. He keeps no references to the array, so it should be collected immediatly on each call to GC.collect(). OR at least, there is no reason for it not to be collected. That said, I'm unsure "GC.collect()" promises it'll acutally *do* anything, it's more of a hint "now would be a good time to run a collection" (AFAIK).

(In reply to comment #0) > import core.memory; > import std.traits; > > void main() { > while(true) { > GC.collect(); > new ubyte[40 * 1024 * 1024]; > } > } > > Application's memory usage grows and it crashes with message: > core.exception.OutOfMemoryError > > If i comment import std.traits; everything goes OK, but now application uses > 300 MM of memory. 32bits? Allocating large blocks in a loop? Good luck with that... sooner or later they would be pinned by something that looks like a pointer to GC. We are back to false pointers and precise collection topic. How std.traits affects that is interesting thing, maybe inflates executable size. Anyway this is game of chance so any slight change may avoid it or bring it back.

(In reply to comment #8) > 32bits? Allocating large blocks in a loop? > Good luck with that... sooner or later they would be pinned by something that > looks like a pointer to GC. We are back to false pointers and precise > collection topic. > > How std.traits affects that is interesting thing, maybe inflates executable > size. Anyway this is game of chance so any slight change may avoid it or bring > it back. 2 things I don't understand: 1. New initializes, so the block contains nothing but 0x00's. How could that contain a false pointers? EDIT: Maybe the size/capacity info is wrongly seen as pointer? 2. Besides, the GC knows about NO_SCAN, doesn't it? We are explicitly asking for new ubytes, which is a non-pointer type. Why would the GC scan any of that at all anywas? If replace the call with GC.malloc(BIG_SIZE, GC.BlkAttr.NO_SCAN), then my win32 runs forever... -------- TLDR, I don't see how the false pointer issue in 32 bit environment would produce this failure.

(In reply to comment #9) > (In reply to comment #8) > > 32bits? Allocating large blocks in a loop? > > Good luck with that... sooner or later they would be pinned by something that > > looks like a pointer to GC. We are back to false pointers and precise > > collection topic. > > > > How std.traits affects that is interesting thing, maybe inflates executable > > size. Anyway this is game of chance so any slight change may avoid it or bring > > it back. > > 2 things I don't understand: > > 1. New initializes, so the block contains nothing but 0x00's. How could that > contain a false pointers? EDIT: Maybe the size/capacity info is wrongly seen as False pointers that HAPPEN TO POINT INTO THAT BLOCK. I can't imagine we are having this conversation again. The larger the block the better target it makes. > pointer? > > 2. Besides, the GC knows about NO_SCAN, doesn't it? We are explicitly asking > for new ubytes, which is a non-pointer type. Why would the GC scan any of that > at all anywas? > Yes it SHOULDN'T scan THAT BLOCK. > If replace the call with GC.malloc(BIG_SIZE, GC.BlkAttr.NO_SCAN), then my win32 > runs forever... > Or change something else and it might not... ? This is an erratic issue. > -------- > > TLDR, I don't see how the false pointer issue in 32 bit environment would > produce this failure.

(In reply to comment #10) > (In reply to comment #9) > > 2 things I don't understand: > > > > 1. New initializes, so the block contains nothing but 0x00's. How could that > > contain a false pointers? EDIT: Maybe the size/capacity info is wrongly seen as > > False pointers that HAPPEN TO POINT INTO THAT BLOCK. I can't imagine we are > having this conversation again. The larger the block the better target it > makes. Ah. Right. I apologize for not being there the last time you had this conversation...

It's array of ubytes, not pointers. GC mustn't scan its content.

(In reply to comment #12) > It's array of ubytes, not pointers. GC mustn't scan its content. I think you misunderstand, something in std.traits gets scanned and it is treated as a pointer into the array of ubytes.

This is due to false pointers as discussed above. Consider the program at the end of this comment: Version = A reproduced the claim. Version !=A uses ~80MiB because of the lower probability of false pointers. ---------- Begin code ----------- import core.memory; import std.traits; void main() { while(true) { GC.collect(); version(A) auto tmp = cast(ubyte*)GC.malloc(40*1024*1024, GC.BlkAttr.NO_SCAN); else auto tmp = cast(ubyte*)GC.malloc(40*1024*1024, GC.BlkAttr.NO_SCAN | GC.BlkAttr.NO_INTERIOR); for (uint i = 0; i < 40*1024*1024; i+= 1024) tmp[i] = 0; // poke the memory tmp = null; } } ---------- End code ----------- *** This issue has been marked as a duplicate of issue 3463 ***