Bug 16641 – Infinite loop on InvalidMemoryOperationError in __dmd_personality_v0

Server seems to get stuck on an InvalidMemoryOperationError stack trace infinite loop every now and then on dmd 2.071. Very rare occurrence but forces me to manually monitor the server with htop (gdb) bt #0 0x0000000001362c13 in __dmd_personality_v0 () #1 0x00007fc83b3d3f43 in ?? () from /lib64/libgcc_s.so.1 #2 0x00007fc83b3d4467 in _Unwind_Resume () from /lib64/libgcc_s.so.1 #3 0x000000000100c3f3 in vibe.stream.botan.BotanTLSStream.finalize() () #4 0x000000000100c099 in vibe.stream.botan.BotanTLSStream.close() () #5 0x0000000001080093 in vibe.http.server.handleHTTPConnection(vibe.core.net.TCPConnection, vibe.http.server.HTTPServerListener) () #6 0x000000000107f4cc in vibe.http.server.listenHTTPPlain(vibe.http.server.HTTPServerSettings).doListen(vibe.http.server.HTTPServerSettings, ulong, immutable(char)[]).__lambda4(vibe.core.net.TCPConnection) () #7 0x0000000000f89f2e in vibe.core.drivers.libasync.LibasyncTCPConnection.onConnect() () #8 0x0000000000f58448 in vibe.core.core.makeTaskFuncInfo!(void() delegate).makeTaskFuncInfo(ref void() delegate).callDelegate(vibe.core.core.TaskFuncInfo*) () #9 0x0000000000f7e65a in vibe.core.core.CoreTask.run() () #10 0x00000000013c052a in core.thread.Fiber.run() () #11 0x00000000013c02c0 in fiber_entryPoint () #12 0x0000000000000000 in ?? () rax 0x7fc83ca3f000 140497987563520 rbx 0x13e59b20 333814560 rcx 0x172ee10 24309264 rdx 0x7fc83ca3f000 140497987563520 rsi 0x172e7e0 24307680 rdi 0x7fc83ca3f000 140497987563520 rbp 0x7fc450b52650 0x7fc450b52650 rsp 0x7fc450b525e0 0x7fc450b525e0 r8 0x0 0 r9 0x0 0 r10 0x7fc450b52690 140481144366736 r11 0xffffffffffffffa8 -88 r12 0x7fc83ca3f0b0 140497987563696 r13 0x1477ff8 21463032 r14 0x7fc83ca3f000 140497987563520 r15 0x7fc83ca3e7c0 140497987561408 rip 0x1362c13 0x1362c13 <__dmd_personality_v0+575> eflags 0x206 [ PF IF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 0x1362bd2 <__dmd_personality_v0+510> mov (%rbx),%rdi x0x1362bd5 <__dmd_personality_v0+513> callq 0x13622c8 <_d_dynamic_cast> x0x1362bda <__dmd_personality_v0+518> mov %rax,%r14 x0x1362bdd <__dmd_personality_v0+521> test %r14,%r14 x0x1362be0 <__dmd_personality_v0+524> je 0x1362bff <__dmd_personality_v0+555> x0x1362be2 <__dmd_personality_v0+526> mov $0x172e7e0,%esi x0x1362be7 <__dmd_personality_v0+531> mov (%r12),%rdi x0x1362beb <__dmd_personality_v0+535> callq 0x13622c8 <_d_dynamic_cast> x0x1362bf0 <__dmd_personality_v0+540> test %rax,%rax x0x1362bf3 <__dmd_personality_v0+543> jne 0x1362bff <__dmd_personality_v0+555> x0x1362bf5 <__dmd_personality_v0+545> mov (%r12),%rsi x0x1362bf9 <__dmd_personality_v0+549> mov %rsi,0x48(%r14) x0x1362bfd <__dmd_personality_v0+553> jmp 0x1362c42 <__dmd_personality_v0+622> x0x1362bff <__dmd_personality_v0+555> mov (%r12),%rdx x0x1362c03 <__dmd_personality_v0+559> cmpq $0x0,0x40(%rdx) x0x1362c08 <__dmd_personality_v0+564> je 0x1362c15 <__dmd_personality_v0+577> -> x0x1362c0a <__dmd_personality_v0+566> mov 0x40(%rdx),%rdx x0x1362c0e <__dmd_personality_v0+570> cmpq $0x0,0x40(%rdx) -> x0x1362c13 <__dmd_personality_v0+575> jne 0x1362c0a <__dmd_personality_v0+566> x0x1362c15 <__dmd_personality_v0+577> mov (%rbx),%rax x0x1362c18 <__dmd_personality_v0+580> mov %rax,0x40(%rdx) x0x1362c1c <__dmd_personality_v0+584> mov (%r12),%rcx x0x1362c20 <__dmd_personality_v0+588> mov %rcx,(%rbx) x0x1362c23 <__dmd_personality_v0+591> mov 0x28(%r12),%r8d (gdb) p *(void**)($rdx+0x40) $8 = (void *) 0x7fc83ca3f000 (gdb) p *(void**)0x7fc83ca3f000 $9 = (void *) 0x172ee10 <vtable for core.exception.InvalidMemoryOperationError> (gdb) p *(void**)(0x7fc83ca3f000+0x40) $10 = (void *) 0x7fc83ca3f000

Do you have code to reproduce this? InvalidMemoryOperationError is usually thrown when a destructor tries to allocate GC memory during a collection. The current GC can't do that. So problem might be in your code or in vibe.d instead of druntime.

(In reply to ag0aep6g from comment #1) > Do you have code to reproduce this? InvalidMemoryOperationError is usually > thrown when a destructor tries to allocate GC memory during a collection. > The current GC can't do that. So problem might be in your code or in vibe.d > instead of druntime. Unfortunately the stack trace cycles into itself so I can't pinpoint the problem, but it always happens here: https://github.com/etcimon/vibe.0/blob/master/source/vibe/stream/botan.d#L147 The issue here happens very very rarely, it's probably a very complex set of circumstances, but the fact here is that I didn't voluntarily create a cycle in the stack trace linked list and the problem is amplified by this function in druntime lacking proper cycle detection.

We got a similar report for dub-registry (not using botan I assume).

I still get this infinite loop problem every now and then. I got around it by auto-launching more processes and killing them every hour, it seems like it happens when my (very busy) application starts to throw more frequently. It became 99% less frequent after putting a loop counter in this code auto eh = ExceptionHeader.toExceptionHeader(exceptionObject); int infini_guard; while (eh.next && ++infini_guard < 10000) { ExceptionHeader* ehn = eh.next;

It tries to throw exception from processException method? You have InvalidMemoryOperationError stored in m_ex field?

I've experienced issue similar to this while working on: https://github.com/dlang/druntime/pull/1872 The trick is that chaining exceptions doesn't work with `InvalidMemoryOperationError` or similar errors that are statically allocated. So, if you're executing your `close()` during stack unwinding due to `AssertError`, say, (like in `scope(exit)`) and that method throws another Error (invalid GC operation), that will reuse the same statically allocated memory, chaining `ThrowableInstance.next = &ThrowableInstance`, which will cause infinite loop in the exception handling.

@schveiguy updated dlang/dmd pull request #14710 "Remove GC for allocating traceinfo" fixing this issue: - Remove GC for allocating traceinfo. Allow traces for InvalidMemoryOperationError and other sensitive errors. Fixes 20650,16641,22616. https://github.com/dlang/dmd/pull/14710

dlang/dmd pull request #14710 "Avoid GC for allocating traceinfo" was merged into master: - 374f20760e6c6183861dc599e1fb6379830a032a by Steven Schveighoffer: Avoid GC for allocating traceinfo. Allow traces for InvalidMemoryOperationError and other sensitive errors. Fixes 20650,16641,22616. https://github.com/dlang/dmd/pull/14710