Bug 18691 – assigning a std.regex.Captures with 3 or more groups causes double free

This minimal test case crashes: martind@swiftboat:~/tmp/D134366$ cat utilimal.d import std.regex; void main() { auto rx = regex("()()()"); auto ma = "".matchFirst(rx); ma = "".matchFirst(rx); } martind@swiftboat:~/tmp/D134366$ ~/download/d/dmd/generated/linux/release/64/dmd -g utilimal.d && valgrind ./utilimal ... ==655== Invalid free() / delete / delete[] / realloc() ==655== at 0x4C29E90: free (vg_replace_malloc.c:473) ==655== by 0x4C1E26: _D3std5regex__T8CapturesTAyaZQo6__dtorMFNbNiNeZv (/home/martind/download/d/dmd/generated/linux/release/64/../../../../../phobos/std/regex/package.d:565) ==655== by 0x48A1CB: _Dmain (utilimal.d:5) ==655== by 0x4C6F5F: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv (in /home/martind/tmp/D134366/utilimal) ==655== by 0x4C6DEF: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv (in /home/martind/tmp/D134366/utilimal) ==655== by 0x4C6ECE: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZv (in /home/martind/tmp/D134366/utilimal) ==655== by 0x4C6DEF: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv (in /home/martind/tmp/D134366/utilimal) ==655== by 0x4C6D5A: _d_run_main (in /home/martind/tmp/D134366/utilimal) ==655== by 0x4C303D: main (in /home/martind/tmp/D134366/utilimal) ==655== Address 0x5d2be50 is 0 bytes inside a block of size 64 free'd ==655== at 0x4C29E90: free (vg_replace_malloc.c:473) ==655== by 0x4C1E26: _D3std5regex__T8CapturesTAyaZQo6__dtorMFNbNiNeZv (/home/martind/download/d/dmd/generated/linux/release/64/../../../../../phobos/std/regex/package.d:565) ==655== by 0x4C2D2F: _D3std5regex__T8CapturesTAyaZQo__T8opAssignZQkMFNbNiNeSQCbQCa__TQBxTQBrZQCfZQw (/home/martind/download/d/dmd/generated/linux/release/64/../../../../../phobos/std/regex/package.d:685) ==655== by 0x48A181: _Dmain (utilimal.d:6) ==655== by 0x4C6F5F: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv (in /home/martind/tmp/D134366/utilimal) ==655== by 0x4C6DEF: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv (in /home/martind/tmp/D134366/utilimal) ==655== by 0x4C6ECE: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZv (in /home/martind/tmp/D134366/utilimal) ==655== by 0x4C6DEF: _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv (in /home/martind/tmp/D134366/utilimal) ==655== by 0x4C6D5A: _d_run_main (in /home/martind/tmp/D134366/utilimal) ==655== by 0x4C303D: main (in /home/martind/tmp/D134366/utilimal) My testing suggests that this is a regression in 2.079.0 over 2.078.3-0. I might risk a guess that it's due to the addition of opAssign to the Captures struct in: https://github.com/dlang/phobos/commit/59520969ef73eaf0691972ee00b389e5bbc4c8fb#diff-4715499b2ff2d74e4eb3c6f3909c611c in an attempt by @MartinNowak to "fix Issue 18114 - regex performance regression". Do we now have big_matches in two Captures objects referring to the same calloc/free memory but each with their own _refcount? Have we also leaked any old memory that (lhs) big_matches owned?

I can't reproduce the crash neither with 2.079 nor with master (in valgrind too). And the same for run.dlang.io: https://run.dlang.io/is/2ncpH0 Is there anything specific about your setup?

How frustrating. I can reproduce it on every one of the three machines I've tried. The output I raised the issue with was from today's master built on Debian Jessie. I found the problem originally on stock 2.079-0 from d-apt on Debian Stretch... like the one below. I was hoping for more symbolic debugging but the -g doesn't seem to affect it for me. mad@shuttle:~/tmp/D134366$ dmd --version DMD64 D Compiler v2.079.0 Copyright (C) 1999-2018 by The D Language Foundation, All Rights Reserved written by Walter Bright mad@shuttle:~/tmp/D134366$ dmd -g utilimal.d && ./utilimal *** Error in `./utilimal': double free or corruption (fasttop): 0x0000563fea6ca140 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7f8c10084bcb] /lib/x86_64-linux-gnu/libc.so.6(+0x76f96)[0x7f8c1008af96] /lib/x86_64-linux-gnu/libc.so.6(+0x777de)[0x7f8c1008b7de] ./utilimal(_D3std5regex__T8CapturesTAyaZQo6__dtorMFNbNiNeZv+0x3b)[0x563fe94fda3f] ./utilimal(_Dmain+0xfd)[0x563fe94c6ea9] ./utilimal(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv+0x28)[0x563fe9502b88] ./utilimal(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv+0x20)[0x563fe9502a18] ./utilimal(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZv+0x8b)[0x563fe9502af7] ./utilimal(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ7tryExecMFMDFZvZv+0x20)[0x563fe9502a18] ./utilimal(_d_run_main+0x1cf)[0x563fe9502983] ./utilimal(main+0x22)[0x563fe94febd6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f8c100342b1] ./utilimal(_start+0x2a)[0x563fe94c6c2a] ======= Memory map: ======== 563fe9426000-563fe9569000 r-xp 00000000 08:07 76285143 /u2/home/mad/tmp/D134366/utilimal 563fe9769000-563fe976a000 r--p 00143000 08:07 76285143 /u2/home/mad/tmp/D134366/utilimal 563fe976a000-563fe979f000 rw-p 00144000 08:07 76285143 /u2/home/mad/tmp/D134366/utilimal 563fea6ca000-563fea6eb000 rw-p 00000000 00:00 0 [heap] 7f8c0c000000-7f8c0c021000 rw-p 00000000 00:00 0 7f8c0c021000-7f8c10000000 ---p 00000000 00:00 0 7f8c10014000-7f8c101a9000 r-xp 00000000 08:01 1859659 /lib/x86_64-linux-gnu/libc-2.24.so 7f8c101a9000-7f8c103a9000 ---p 00195000 08:01 1859659 /lib/x86_64-linux-gnu/libc-2.24.so 7f8c103a9000-7f8c103ad000 r--p 00195000 08:01 1859659 /lib/x86_64-linux-gnu/libc-2.24.so 7f8c103ad000-7f8c103af000 rw-p 00199000 08:01 1859659 /lib/x86_64-linux-gnu/libc-2.24.so 7f8c103af000-7f8c103b3000 rw-p 00000000 00:00 0 7f8c103b3000-7f8c103c9000 r-xp 00000000 08:01 1859854 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f8c103c9000-7f8c105c8000 ---p 00016000 08:01 1859854 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f8c105c8000-7f8c105c9000 r--p 00015000 08:01 1859854 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f8c105c9000-7f8c105ca000 rw-p 00016000 08:01 1859854 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f8c105ca000-7f8c105cd000 r-xp 00000000 08:01 1859704 /lib/x86_64-linux-gnu/libdl-2.24.so 7f8c105cd000-7f8c107cc000 ---p 00003000 08:01 1859704 /lib/x86_64-linux-gnu/libdl-2.24.so 7f8c107cc000-7f8c107cd000 r--p 00002000 08:01 1859704 /lib/x86_64-linux-gnu/libdl-2.24.so 7f8c107cd000-7f8c107ce000 rw-p 00003000 08:01 1859704 /lib/x86_64-linux-gnu/libdl-2.24.so 7f8c107ce000-7f8c107d5000 r-xp 00000000 08:01 1859828 /lib/x86_64-linux-gnu/librt-2.24.so 7f8c107d5000-7f8c109d4000 ---p 00007000 08:01 1859828 /lib/x86_64-linux-gnu/librt-2.24.so 7f8c109d4000-7f8c109d5000 r--p 00006000 08:01 1859828 /lib/x86_64-linux-gnu/librt-2.24.so 7f8c109d5000-7f8c109d6000 rw-p 00007000 08:01 1859828 /lib/x86_64-linux-gnu/librt-2.24.so 7f8c109d6000-7f8c10ad9000 r-xp 00000000 08:01 1859707 /lib/x86_64-linux-gnu/libm-2.24.so 7f8c10ad9000-7f8c10cd8000 ---p 00103000 08:01 1859707 /lib/x86_64-linux-gnu/libm-2.24.so 7f8c10cd8000-7f8c10cd9000 r--p 00102000 08:01 1859707 /lib/x86_64-linux-gnu/libm-2.24.so 7f8c10cd9000-7f8c10cda000 rw-p 00103000 08:01 1859707 /lib/x86_64-linux-gnu/libm-2.24.so 7f8c10cda000-7f8c10cf2000 r-xp 00000000 08:01 1859816 /lib/x86_64-linux-gnu/libpthread-2.24.so 7f8c10cf2000-7f8c10ef1000 ---p 00018000 08:01 1859816 /lib/x86_64-linux-gnu/libpthread-2.24.so 7f8c10ef1000-7f8c10ef2000 r--p 00017000 08:01 1859816 /lib/x86_64-linux-gnu/libpthread-2.24.so 7f8c10ef2000-7f8c10ef3000 rw-p 00018000 08:01 1859816 /lib/x86_64-linux-gnu/libpthread-2.24.so 7f8c10ef3000-7f8c10ef7000 rw-p 00000000 00:00 0 7f8c10ef7000-7f8c10f1a000 r-xp 00000000 08:01 1859630 /lib/x86_64-linux-gnu/ld-2.24.so 7f8c10fe6000-7f8c110eb000 rw-p 00000000 00:00 0 7f8c11116000-7f8c1111a000 rw-p 00000000 00:00 0 7f8c1111a000-7f8c1111b000 r--p 00023000 08:01 1859630 /lib/x86_64-linux-gnu/ld-2.24.so 7f8c1111b000-7f8c1111c000 rw-p 00024000 08:01 1859630 /lib/x86_64-linux-gnu/ld-2.24.so 7f8c1111c000-7f8c1111d000 rw-p 00000000 00:00 0 7ffe6f231000-7ffe6f252000 rw-p 00000000 00:00 0 [stack] 7ffe6f3f4000-7ffe6f3f6000 r--p 00000000 00:00 0 [vvar] 7ffe6f3f6000-7ffe6f3f8000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Aborted mad@shuttle:~/tmp/D134366$ Perhaps Debian's libc has more debugging configured than others but surely valgrind ought to behave roughly the same: mad@shuttle:~/tmp/D134366$ valgrind --version valgrind-3.12.0.SVN mad@shuttle:~/tmp/D134366$

If you're interested in this bug, then you're likely to also be interested in: Issue 18692: assignment of std.regex.Captures reads freed memory from 2.072.0 to 2.078.3 inclusive

*** Issue 18692 has been marked as a duplicate of this issue. ***

This is the same issue - bogus CoW implementation. PR is out

Commits pushed to master at https://github.com/dlang/phobos https://github.com/dlang/phobos/commit/b602b5a3f42456faa39b56082510d2181b6b844a Fix issue 18691 - memory errors in std.regex.Captures Instead of a messy code that ended up with refcount at the wrong place provide a clean fixed-size array with required semantics and build on top of that. https://github.com/dlang/phobos/commit/71692cd4746bc9601c3126666a71eb8b20fd22f3 Merge pull request #6379 from DmitryOlshansky/fix-issue-18691 Fix issue 18691 - memory errors in std.regex.Captures merged-on-behalf-of: Jack Stouffer <[email protected]>