Bug 14894 – mangling of mixins and lambdas is not unique and depends on compilation flags

Just found this issue while testing dub. Generating test runner configuration '__test__library__' for 'library' (library). Performing "unittest" build using dmd for x86_64. Building dub 0.9.24-rc.3+commit.9.g5f397bc configuration "__test__library__"... Linking... source/dub/internal/utils.d:207: error: undefined reference to '_D3std3net4curl4HTTP9__mixin3610verifyPeerMFNdbZv' source/dub/internal/utils.d:210: error: undefined reference to '_D3std3net4curl4HTTP9__mixin365proxyMFNdAxaZv' /usr/include/dmd/phobos/std/net/curl.d:295: error: undefined reference to '_D3std3net4curl4HTTP9__mixin369onReceiveMFNdDFAhZmZv' /usr/include/dmd/phobos/std/net/curl.d:773: error: undefined reference to '_D3std3net4curl4HTTP9__mixin369onReceiveMFNdDFAhZmZv' /usr/include/dmd/phobos/std/net/curl.d:786: error: undefined reference to '_D3std3net4curl4HTTP9__mixin366onSendMFNdDFAvZmZv' /usr/include/dmd/phobos/std/net/curl.d:786: error: undefined reference to '_D3std3net4curl4HTTP9__mixin366handleMFNcNdNjZS3std3net4curl4Curl' /usr/include/dmd/phobos/std/net/curl.d:762: error: undefined reference to '_D3std3net4curl4HTTP9__mixin369onReceiveMFNdDFAhZmZv' /usr/include/dmd/phobos/std/net/curl.d:769: error: undefined reference to '_D3std3net4curl4HTTP9__mixin366onSendMFNdDFAvZmZv' /usr/include/dmd/phobos/std/net/curl.d:769: error: undefined reference to '_D3std3net4curl4HTTP9__mixin366handleMFNcNdNjZS3std3net4curl4Curl' The mixin is named _D3std3net4curl4HTTP9__mixin35 (35 instead of 36) in libphobos2.a. Not sure why it passes on travis-ci (https://travis-ci.org/D-Programming-Language/dub/jobs/74832612).

Happens b/c the mangling is determined by the number of member in the parent scope. This number can vary between compilations w/ and w/o -unittest. cat > main.d << CODE import bug; void main() { Foo foo; foo.onReceive(); } CODE cat > bug.d << CODE mixin template Protocol() { void onReceive() {} } struct Foo { mixin Protocol!(); unittest { } } CODE ---- dmd -c bug dmd -unittest main bug.o ---- The manifestation of this bug is caused by skipping unittest parsing [¹], but the underlying problem can also be triggered by any other version/debug block member. The code to derive a unique mixin mangling was already added with 2.065.0 [²] and is also used for lambdas. [¹]: https://github.com/D-Programming-Language/dmd/pull/4704 [²]: https://github.com/D-Programming-Language/dmd/pull/3019

Commits pushed to stable at https://github.com/D-Programming-Language/dmd https://github.com/D-Programming-Language/dmd/commit/d10bd6c88426b3566c25f42d7194bf5c98a90a38 workaround Issue 14894 - linkage error with mixin template in std.net.curl https://github.com/D-Programming-Language/dmd/commit/24214150e3463c94709f55c4ade912b974bbf911 Merge pull request #4871 from MartinNowak/workaround14894 workaround Issue 14894 - linkage error with mixin template in std.net.curl

We still need a long-term solution to make the mangling not dependent on flags like -unittest or -release. Using source position (line and col) doesn't work b/c the position might differ, e.g. in a .di file or when editing a comment.

Commits pushed to master at https://github.com/D-Programming-Language/dmd https://github.com/D-Programming-Language/dmd/commit/d10bd6c88426b3566c25f42d7194bf5c98a90a38 workaround Issue 14894 - linkage error with mixin template in std.net.curl https://github.com/D-Programming-Language/dmd/commit/24214150e3463c94709f55c4ade912b974bbf911 Merge pull request #4871 from MartinNowak/workaround14894

(In reply to Martin Nowak from comment #4) > We still need a long-term solution to make the mangling not dependent on > flags like -unittest or -release. > Using source position (line and col) doesn't work b/c the position might > differ, e.g. in a .di file or when editing a comment. This is very similar to the -debug issues we have with template instantiation. The bottom line would be that only libs with the same flags are linkable. BTW, g++ counts the number of lambdas per function and assigns them an increasing id. Using separate counters per scope per entity at least mitigates the issue.

*** Issue 16994 has been marked as a duplicate of this issue. ***

Apparently can also cause calls of the wrong function instead of linker errors, see issue 16994. Different compilations end up with instances of the same mangling. cat > lib.d << CODE alias min = (a, b) => a < b ? a : b; alias max = (a, b) => a > b ? a : b; auto reduce(alias func, T...)(T args) { return func(args); } CODE cat > a.d << CODE import lib; unittest { assert(reduce!min(1, 2) == 1); } CODE cat > b.d << CODE import lib; unittest { assert(reduce!max(1, 2) == 2); } CODE # works dmd -main -unittest -ofmain a.d b.d ./main # fails dmd -unittest -c a.d dmd -unittest -c b.d dmd -main -ofmain a.o b.o // one of the tests calls the wrong reduce ./main ---- $ dmd -unittest -c a.d $ nm a.o | grep lambda 0000000000000000 W _D3lib18__T9__lambda5TiTiZ9__lambda5FNaNbNiNfiiZi 0000000000000000 W _D3lib32__T6reduceS143lib9__lambda5TiTiZ6reduceFNaNbNiNfiiZi $ dmd -unittest -c b.d $ nm b.o | grep lambda 0000000000000000 W _D3lib18__T9__lambda5TiTiZ9__lambda5FNaNbNiNfiiZi 0000000000000000 W _D3lib32__T6reduceS143lib9__lambda5TiTiZ6reduceFNaNbNiNfiiZi

The way I see forward is to not use a number. But to disambiguate by a reproducible hash.

(In reply to uplink.coder from comment #9) > The way I see forward is to not use a number. > But to disambiguate by a reproducible hash. Yes! What to hash though? Source code, tokens, parsed AST? Guess the latter would remain identical even after reformatting which is a nice property (but not an important one).

(In reply to Martin Nowak from comment #10) > (In reply to uplink.coder from comment #9) > > The way I see forward is to not use a number. > > But to disambiguate by a reproducible hash. > > Yes! What to hash though? Source code, tokens, parsed AST? Guess the latter > would remain identical even after reformatting which is a nice property (but > not an important one). We need AST-Node hashing anyhow. but that won't fix the .di files problem, as the hash will be different from the impl.... or does the source have to be available ? If it does then we stand a chance. This will also auto-magically de-duplicate lambdas with the same bodys.

(In reply to uplink.coder from comment #11) > We need AST-Node hashing anyhow. > but that won't fix the .di files problem, as the hash will be different from > the impl.... or does the source have to be available ? Yes, source code for lambdas and mixins should be available in situations where we use such numbers, so either of them could be hashed. Literal source code is only available during parsing, so hashes would need to be computed eagerly during the performance sensitive lexing. A hash on the AST could be computed lazily. We already have `expressionHash` in dmd, but that depends on allocation addresses.

THIS ISSUE HAS BEEN MOVED TO GITHUB https://github.com/dlang/dmd/issues/19023 DO NOT COMMENT HERE ANYMORE, NOBODY WILL SEE IT, THIS ISSUE HAS BEEN MOVED TO GITHUB