Bug 14828 – [REG2.067] duplicate symbol __ModuleInfoZ depending on ordering of files passed to dmd

Reduced test case: see https://github.com/timotheecour/dsnippet/tree/master/bug_duplicate_symbol_error The makefile in that directory should explain the bug: Makefile: ---- dmd=~/git_clone/D/Digger/result/bin/dmd #works with dmd all0: ${dmd} -oflibproj6.a -lib -g project2.d foo2.d foo1.d ${dmd} -ofout -g libproj6.a foo1.d main.d #works with dmd all1: ${dmd} -oflibproj6.a -lib -g project2.d foo1.d foo2.d ${dmd} -ofout -g libproj6.a foo2.d main.d #worked with dmd_067_1_X #works with dmd from: ./digger build v2.068.0-b1 #FAILS with dmd from: ./digger build #FAILS with dmd from: ./digger build master (v2.068-devel-494a91d) #has failed for a few months on git master AFAIR all1_order_flipped: ${dmd} -oflibproj6.a -lib -g project2.d foo2.d foo1.d ${dmd} -ofout -g libproj6.a foo2.d main.d ---- This bug is very annoying as it prevents partial recompilation

digger bisect returns: 2015-07-26T13:16:37.992:common.d:log:25 digger: bisect.d:113 ba17fa7669885dca1c8f7b246483149d37b3f10d is the first bad commit commit ba17fa7669885dca1c8f7b246483149d37b3f10d Author: Martin Nowak <[email protected]> Date: Tue Mar 31 04:43:51 2015 +0200 phobos: Merge pull request #2956 from jmdavis/tempFile https://github.com/D-Programming-Language/phobos/pull/2956 Implement issue# 13996. Add File.tempFile. Although this may just be a side effect of another root cause?

using different parameters for digger bisect: bad = master @ 1 month ago good = master @ 2 months ago returns: digger: bisect.d:113 f1bf3d208546728d433de7b3d53631497846b117 is the first bad commit commit f1bf3d208546728d433de7b3d53631497846b117 Author: Dmitry Olshansky <[email protected]> Date: Fri Jun 26 20:08:27 2015 +0300 phobos: Merge pull request #3443 from CyberShadow/pull-20150626-091553-uni-stray-stdio-import https://github.com/D-Programming-Language/phobos/pull/3443 std.uni: Remove stray std.stdio import

Is the root cause a symbol that should've been a weak symbol?

Equivalent link failure happens in win32/64. But the regression version is not same: With 2.064, 2.065, 2.066.0, 2.066.1 --> no problem happens With 2.067.0, 2.067.1, 2.068.0-beta1, 2.068.0-beta2, git-head --> duplicated symbol error happens with 'make all1_order_flipped'

(In reply to Timothee Cour from comment #0) > This bug is very annoying as it prevents partial recompilation There are many catches with that.

This might have the same cause as issue 14748.

Created attachment 1534 Win32 reduced case without Phobos dependency (In reply to Kenji Hara from comment #4) > Equivalent link failure happens in win32/64. > But the regression version is not same: > > With 2.064, 2.065, 2.066.0, 2.066.1 > --> no problem happens > > With 2.067.0, 2.067.1, 2.068.0-beta1, 2.068.0-beta2, git-head > --> duplicated symbol error happens with 'make all1_order_flipped' In win32, the attached reduced case fails from 2.067. It was introduced in: > commit 679929fa2da120db76d91303efd2316370970d1e > Merge: 430bdba 706aa61 > Author: Andrei Alexandrescu <[email protected]> > Date: Wed Jan 28 08:44:07 2015 -0800 > > Merge pull request #4354 from 9rnsr/fix14044 > > [REG2.066] Issue 14044 - dmd generates spurious functions in object file created from empty module However, I don't understand why the change introduced the bug... Furthermore, I'm not sure that's really same with the original issue, because it happens from 2.068-beta2 in OSX.

(In reply to Timothee Cour from comment #3) > Is the root cause a symbol that should've been a weak symbol? ModuleInfos aren't weak symbols. You didn't past the error message. Which ModuleInfo is duplicated? I guess it's either foo2 or main.

(In reply to Martin Nowak from comment #8) > (In reply to Timothee Cour from comment #3) > > Is the root cause a symbol that should've been a weak symbol? > > ModuleInfos aren't weak symbols. > You didn't past the error message. Which ModuleInfo is duplicated? > I guess it's either foo2 or main. duplicate symbol _D4foo212__ModuleInfoZ in: out.o libproj6.a(foo2.o) ld: 1 duplicate symbol for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation)

(In reply to Martin Nowak from comment #5) > (In reply to Timothee Cour from comment #0) > > This bug is very annoying as it prevents partial recompilation > > There are many catches with that. I'm well aware. However this technique, when properly used, allows very fast recompilation. We should not prevent this.

(In reply to Timothee Cour from comment #9) > duplicate symbol _D4foo212__ModuleInfoZ in: > out.o > libproj6.a(foo2.o) > ld: 1 duplicate symbol for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see > invocation) The duplicated symbol name is same with in win32.

Created attachment 1538 A little updated test case OK, I understand the issue mechanism. It's obvious from the list files of libproj.lib and out.obj. (`make genlist`) With the `make a1f` command, the last link for out.exe will pull foo2.obj in the lib file, because: The instantiated function _D5stdio12__T7writelnZ7writelnFNaNbNiNfAyaZv uses _D5stdio7__arrayZ for array bounds check, but it's stored in foo2.obj in the lib. By that, the symbol _D4foo212__ModuleInfoZ in lib will also linked because it's in foo2.obj. On the other hand, out.obj also contains _D4foo212__ModuleInfoZ, then they conflicts each other. ---- The essential fix I think is: when the lib file is compiled, the helper functions (__array, etc) of non-root modules (e.g. stdio) should be separated their own obj file.

The module info symbol should only be generated by compiling foo2 from the command line. Why is it generated into out.obj?

(In reply to Walter Bright from comment #13) > The module info symbol should only be generated by compiling foo2 from the > command line. Why is it generated into out.obj? foo2.d is compiled twice. One generated obj is stored into the lib file, one another is stored into out.obj. This is the issue case from the original case. all1_order_flipped: ${dmd} -oflibproj6.a -lib -g project2.d foo2.d foo1.d ${dmd} -ofout -g libproj6.a foo2.d main.d

(In reply to Kenji Hara from comment #12) > The essential fix I think is: when the lib file is compiled, the helper > functions > (__array, etc) of non-root modules (e.g. stdio) should be separated their > own obj file. Implemented. https://github.com/D-Programming-Language/dmd/pull/4851

Different fix: https://github.com/D-Programming-Language/dmd/pull/4859

Commits pushed to stable at https://github.com/D-Programming-Language/dmd https://github.com/D-Programming-Language/dmd/commit/679275df8615740fcc1b1f8efb064ef0174e9d6d fix Issue 14828 - duplicate symbol __ModuleInfoZ - always emit __arrayZ/__assertZ/__unittest_failZ helpers (even in release) to avoid linkage issues when instantiating a template of that module with different compiler flags - also fixes Issue 14748 - Removing std.stdio import causes 2x increase in "Hello, world" same underlying issue, some undefined helper function drag in a different module which happens to have weak definitions of those helpers https://github.com/D-Programming-Language/dmd/commit/2e15e16658a0416003e9fcf9ed7dfcd45e0da844 Merge pull request #4858 from MartinNowak/fix14828 fix Issue 14828 - duplicate symbol __ModuleInfoZ

The fix that was actually pulled: https://github.com/D-Programming-Language/dmd/pull/4858

Commits pushed to master at https://github.com/D-Programming-Language/dmd https://github.com/D-Programming-Language/dmd/commit/679275df8615740fcc1b1f8efb064ef0174e9d6d fix Issue 14828 - duplicate symbol __ModuleInfoZ https://github.com/D-Programming-Language/dmd/commit/2e15e16658a0416003e9fcf9ed7dfcd45e0da844 Merge pull request #4858 from MartinNowak/fix14828

Commit pushed to stable at https://github.com/D-Programming-Language/dlang.org https://github.com/D-Programming-Language/dlang.org/commit/7c62ea0a99aac8c0d74d73ff2d2d6d06b43200a3 warning for Issue 14828 fix

Commit pushed to master at https://github.com/D-Programming-Language/dlang.org https://github.com/D-Programming-Language/dlang.org/commit/7c62ea0a99aac8c0d74d73ff2d2d6d06b43200a3 warning for Issue 14828 fix