Bug 14578 – [ddemangle] core.exception.InvalidMemoryOperationError@(0) handling large symbol list

Created attachment 1522 mangled symbols list Attached is a list of symbols pulled from the phobos unittester. ddemangle should be able to handle all of them, but errors somewhere after the first 125 with: core.exception.InvalidMemoryOperationError@(0) I don't think this is to do with core.demangle per say, just a memory bug in ddemangle itself. Reproducible on Linux x86_64 with ddemangle shipped with 2.067

Fails on _D3std3uni53__T11TrieBuilderTbTuTS3std3uni14__T5clampVmi7Z5clampZ11TrieBuilder8putRangeMFNeuubZ12__dgliteral4MFNaNbNiNfZAxa while trying to decode the function type of FNeuubZ12__dgliteral4MFNaNbNiNfZAxa or so.

Can you get a stack trace? http://wiki.dlang.org/InvalidMemoryOperationError

(gdb) bt #0 onInvalidMemoryOperationError (pretend_sideffect=0x0) at src/core/exception.d:536 #1 0x00000000004dd558 in _d_arrayshrinkfit (ti=0x5275f0 <TypeInfo_Aa.init$>, arr=...) at src/rt/lifetime.d:684 #2 0x00000000004c6906 in object.assumeSafeAppend!(char).assumeSafeAppend(ref inout(char[])) (arr=0x773518) at ./../../src/druntime/import/object.di:662 #3 0x00000000004c5f5e in std.stdio.File.ByLineImpl!(char, char).ByLineImpl.popFront() (this=0x773500) at ./../../src/phobos/std/stdio.d:1677 #4 0x00000000004c5d1b in std.stdio.File.ByLine!(char, char).ByLine.popFront() (this=0x7fffffffda08) at ./../../src/phobos/std/stdio.d:1624 #5 0x00000000004a62be in D main (args=...) at ddemangle.d:79 (gdb) up #1 0x00000000004dd558 in _d_arrayshrinkfit (ti=0x5275f0 <TypeInfo_Aa.init$>, arr=...) at src/rt/lifetime.d:684 (gdb) p info $1 = {base = 0x7ffff7ee8400, size = 1024, attr = 10} (gdb) p newsize $2 = 1023 (gdb) p tinext $3 = (object.TypeInfo *) 0x5279c0 <TypeInfo_a.init$> Going to upgrade this to regression: https://github.com/D-Programming-Language/druntime/commit/0b72987df17354c83bc302672f4f132fe03db65a https://github.com/D-Programming-Language/druntime/commit/e25cab10d763b36b9c63c62e3713134a137e2760 It throws because newsize + 2 overflows 1024, so assumeSafeAppend fails.

OK, got a reduced test: --- import std.stdio; void main() { foreach (line; stdin.byLine()) { } } --- Only require two lines of stdin with the following constraints: #1 Line: Length >= 510 && Length <= 1021 #2 Line: Length == 1023

(In reply to Iain Buclaw from comment #4) > OK, got a reduced test: > > --- > import std.stdio; > void main() > { > foreach (line; stdin.byLine()) > { > } > } > --- > > Only require two lines of stdin with the following constraints: > > #1 Line: Length >= 510 && Length <= 1021 > #2 Line: Length == 1023 Duplicate of issue 14005 and in turn of issue 13856?

(In reply to Iain Buclaw from comment #3) > Going to upgrade this to regression: > https://github.com/D-Programming-Language/druntime/commit/ > 0b72987df17354c83bc302672f4f132fe03db65a > https://github.com/D-Programming-Language/druntime/commit/ > e25cab10d763b36b9c63c62e3713134a137e2760 > > It throws because newsize + 2 overflows 1024, so assumeSafeAppend fails. I don't get what those changes have to do with the last sentence.

(In reply to Walter Bright from comment #6) > (In reply to Iain Buclaw from comment #3) > > Going to upgrade this to regression: > > https://github.com/D-Programming-Language/druntime/commit/ > > 0b72987df17354c83bc302672f4f132fe03db65a > > https://github.com/D-Programming-Language/druntime/commit/ > > e25cab10d763b36b9c63c62e3713134a137e2760 > > > > It throws because newsize + 2 overflows 1024, so assumeSafeAppend fails. > > I don't get what those changes have to do with the last sentence. __setArrayAllocLength returns false if newlength+padding exceeds the size allocated to the array. Before, it just blissfully ignored this assertion and let the program continue. What's interesting in the context of this failure, however, is that newlength == oldlength.

(In reply to Iain Buclaw from comment #4) > OK, got a reduced test: > > --- > import std.stdio; > void main() > { > foreach (line; stdin.byLine()) > { > } > } > --- > > Only require two lines of stdin with the following constraints: > > #1 Line: Length >= 510 && Length <= 1021 > #2 Line: Length == 1023 I can't reproduce this. I made a file with 2 lines, one is 520 bytes, the other is 1023 bytes (I tried adding and subtracting one character to the second line in case you were including newlines in that requirement). Still works without asserting. This is with 2.067 on OSX.

I think this might be a duplicate of https://issues.dlang.org/show_bug.cgi?id=13856 I still cannot reproduce this.

I finally reproduced with 2.067 and 2.067.1, it's definitely Linux specific. However, this is fixed in the HEAD version (at least using your reduced test case). Can you confirm whether this bug is fixed in HEAD for you, Iain?

(In reply to Steven Schveighoffer from comment #10) > I finally reproduced with 2.067 and 2.067.1, it's definitely Linux specific. > However, this is fixed in the HEAD version (at least using your reduced test > case). Can you confirm whether this bug is fixed in HEAD for you, Iain? Did you test the reduced test with: x * 512 times x * 1023 times Just making sure. :)

I created 3 test cases, one based on your recommendation, one based on Ali's in issue 14005, and one based on the post in http://forum.dlang.org/post/[email protected]. In all three cases, they were passing on my Mac. However, running on my Linux VM, they all failed (Frustratingly, nobody attached any exact files, so I wasn't sure if my hand-made files were correctly set up to trigger the bug when I tested on the Mac). Then I cloned and built everything from HEAD in git. All of them passed. Does this mean the problem is fixed? For byLine, I think it is. However, I think readln is still buggy. issue 13856 should remain open until that is fixed, but I think we can close this one. Note, the use of assumeSafeAppend in readln is/was incorrect, it's ignoring the metadata of the block, and blindly assuming it can consume all the data in the block as returned by the GC. So the issue is not with assumeSafeAppend I think.

(In reply to Steven Schveighoffer from comment #12) > I created 3 test cases, one based on your recommendation, one based on Ali's > in issue 14005, and one based on the post in > http://forum.dlang.org/post/[email protected]. In all > three cases, they were passing on my Mac. However, running on my Linux VM, > they all failed (Frustratingly, nobody attached any exact files, so I wasn't > sure if my hand-made files were correctly set up to trigger the bug when I > tested on the Mac). > > Then I cloned and built everything from HEAD in git. All of them passed. > > Does this mean the problem is fixed? For byLine, I think it is. However, I > think readln is still buggy. issue 13856 should remain open until that is > fixed, but I think we can close this one. > > Note, the use of assumeSafeAppend in readln is/was incorrect, it's ignoring > the metadata of the block, and blindly assuming it can consume all the data > in the block as returned by the GC. So the issue is not with > assumeSafeAppend I think. OK. I still haven't had the time to test, but if you build ddemangle using phobos/HEAD and run it on attachment 1522. If all symbols are correctly demangled with no SEGV then this can be closed.

Yes, it works with no SEGV, tested on Linux 64-bit. Confirmed that standard build of ddemangle crashes.