Bug 5326 – [GC] GC -- 99% CPU in gc@gcx@Gcx@mark()

I was running a piece of code that allocated a huge block of memory at once, then allocated small chunks after that inside a block, deleting them at the end of the scope. I managed to get the code down to this: auto temp = new void[0x8080 * 4096]; for (int i = 0; i < 0x8080 * 4; i++) { scope auto temp2 = new void[0x10100]; } What happens is, after the first few ten thousands of collections (for me, it was around after 62,000 collections), the time spent in the mark() method shoots up to around 99% of the application's time, and the program effectively hangs at the allocation. The numbers above were specific to my system; they're rather arbitrary. All you need to do is: (1) Run the code, and break into the process before around 1 second. Notice that, when you do this, it's often inside memset() or some other function, and not the GC. (2) Run the code, and break into the process after several seconds have passed. At this time, it's practically guaranteed that you're in the mark() method -- and in fact, for me, it froze my application.

That's somewhat excpected. The mark() function is the core of the garbage collector, where it basically follows all references in the program's heap. void[] arrays can potentially contain references, so the GC has to mark() these as well. (Why are you using void[]? 10 dollars on you have no real reason.) What makes things worse: each time there's a pointer into a large memory block, the GC has to do a linear scan to find the beginning of the memory block. And you are alocating LARGE blocks (32k pages!), so that might become significant. I think what's happening is that your heap is exploding due to false pointers. The memory allocations are so large that there's always something that looks like a pointer pointing into it, so it never gets free'd. A larger heap means higher collection times. So, this is probably not a bug.

PS: scope doesn't work on arrays. Your example code never explicitly free's the memory.

> PS: scope doesn't work on arrays. Your example code never explicitly free's the memory. That explains everything... but then, how do we delete arrays? > void[] arrays can potentially contain references, so the GC has to mark() these as well. I had NO IDEA that was the case!! (see below) > (Why are you using void[]? 10 dollars on you have no real reason.) I actually did have a reason, but I was misunderstanding the situation. I used void[]'s in places where I did not want ubyte[]s to be interpreted as numbers, so that they would not be dereferenceable. (To me, a void[] could contain binary *data*, whereas a ubyte[] logically means that it contains numbers -- and while it's OK to reference a number in a list, it's *not* OK to me to just randomly reference an arbitrary block of data, which might contain code inside a buffer.) Of course, I wasn't understanding their meaning correctly, so I won't do that anymore; thank you very much for the explanation!!

(In reply to comment #3) > > PS: scope doesn't work on arrays. Your example code never explicitly free's the > memory. > > That explains everything... but then, how do we delete arrays? You can free heap allocated arrays by "delete somearray;" (deprecated in D2). I'm not sure what's the proper way of deleting an array is. As you can see in the implementation for "delete <array>;", it's not that trivial: http://www.dsource.org/projects/druntime/browser/trunk/src/rt/lifetime.d#L937 PS: That code may contain a bug. It may pass a pointer to gc_free() that is not the start of a memory block, but gc_free() could have trouble with that and have undefined effects on the GC heap integrity, but I digress. When you allocate a class using scope on classes, it is normally allocated on the stack. Doing this with arrays is hard; you need to use alloca() and turn the resulting void* into an array... Also, the idea with void[] was probably that it could contain *anything* that can exist in memory. Thus the GC has to assume that it may contain pointers.

Ah, all right, thank you for the information!

Mass migration of bugs marked as x86-64 to just x86. The platform run on isn't what's relevant, it's if the app is a 32 or 64 bit app.

This report has insufficient information. I believe it is a duplicate of #3463.