Bug 24129 – ImportC: MS-Link cannot handle multiple COMDATs with the same name

fatal error LNK1179: invalid or corrupt file: duplicate COMDAT '__acrt_locale_get_ctype_array_value' with DMD64 on Windows - it seems things are NOT being properly inlined. This is likely a new problem with the current MSVC header, because almost any non-trivial program that ends up including ctypes.h (or many other haders) cannot be linked, so nearly nothing can be built with ImportC. Reproducer: DMD version: DMD64 D Compiler v2.105.0-dirty Copyright (C) 1999-2023 by The D Language Foundation, All Rights Reserved written by Walter Bright (NOTE: this is the DMD installer from the website, not sure why it reports as dirty - I did not build it myself). Windows 11 with MSVC Community edition. Here is a minimal reproducer: >>>TYPE a.c b.c a.c #include <ctype.h> extern void xa(void) { } b.c #include <ctype.h> int main(void) { void xa(void); xa(); return 0; } >>>dmd -v a.c b.c predefs DigitalMars LittleEndian D_Version2 all D_SIMD Windows Win64 CRuntime_Microsoft CppRuntime_Microsoft D_InlineAsm_X86_64 X86_64 D_LP64 assert D_PreConditions D_PostConditions D_Invarian ts D_ModuleInfo D_Exceptions D_TypeInfo D_HardFloat binary dmd version v2.105.0-dirty config C:\D\dmd2\windows\bin64\sc.ini DFLAGS -IC:\D\dmd2\windows\bin64\..\..\src\phobos -IC:\D\dmd2\windows\bin64\..\..\src\druntime\import include C:\D\dmd2\windows\bin64\..\..\src\druntime\import\importc.h C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.exe /P /Zc:preprocessor /PD /nologo a.c /FIC:\D\dmd2\windows\bin64\..\..\src\druntime\import\importc.h /Fia.i include C:\D\dmd2\windows\bin64\..\..\src\druntime\import\importc.h C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.exe /P /Zc:preprocessor /PD /nologo b.c /FIC:\D\dmd2\windows\bin64\..\..\src\druntime\import\ importc.h /Fib.i predefs DigitalMars LittleEndian D_Version2 all D_SIMD Windows Win64 CRuntime_Microsoft CppRuntime_Microsoft D_InlineAsm_X86_64 X86_64 D_LP64 assert D_PreConditions D_PostConditions D_Invarian ts D_ModuleInfo D_Exceptions D_TypeInfo D_HardFloat binary dmd version v2.105.0-dirty config C:\D\dmd2\windows\bin64\sc.ini DFLAGS -IC:\D\dmd2\windows\bin64\..\..\src\phobos -IC:\D\dmd2\windows\bin64\..\..\src\druntime\import include C:\D\dmd2\windows\bin64\..\..\src\druntime\import\importc.h C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.exe /P /Zc:preprocessor /PD /nologo a.c /FIC:\D\dmd2\windows\bin64\..\..\src\druntime\import\importc.h /Fia.i include C:\D\dmd2\windows\bin64\..\..\src\druntime\import\importc.h C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\cl.exe /P /Zc:preprocessor /PD /nologo b.c /FIC:\D\dmd2\windows\bin64\..\..\src\druntime\import\importc.h /Fib.i parse a parse b importall a import __builtins (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\__builtins.di) import object (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\object.d) import core.internal.hash (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\hash.d) import core.internal.traits (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\traits.d) import core.internal.entrypoint (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\entrypoint.d) import core.internal.array.appending (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\array\appending.d) import core.internal.array.comparison (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\array\comparison.d) import core.internal.array.equality (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\array\equality.d) import core.internal.array.casting (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\array\casting.d) import core.internal.array.concatenation (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\array\concatenation.d) import core.internal.array.construction (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\array\construction.d) import core.internal.array.arrayassign (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\array\arrayassign.d) import core.internal.array.capacity (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\array\capacity.d) import core.internal.dassert (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\dassert.d) import core.atomic (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\atomic.d) import core.internal.attributes (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\attributes.d) import core.internal.atomic (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\atomic.d) import core.internal.destruction (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\destruction.d) import core.internal.moving (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\moving.d) import core.internal.postblit (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\postblit.d) import core.internal.switch_ (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\internal\switch_.d) import core.lifetime (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\lifetime.d) import core.builtins (C:\D\dmd2\windows\bin64\..\..\src\druntime\import\core\builtins.d) importall b semantic a semantic b semantic2 a semantic2 b semantic3 a semantic3 b inline scan a inlined a.__acrt_locale_get_ctype_array_value => a._chvalidchk_l inlined a.__acrt_locale_get_ctype_array_value => a._chvalidchk_l inlined a._chvalidchk_l => a._ischartype_l inline scan b inlined b.__acrt_locale_get_ctype_array_value => b._chvalidchk_l inlined b.__acrt_locale_get_ctype_array_value => b._chvalidchk_l inlined b._chvalidchk_l => b._ischartype_l code a function a.__acrt_locale_get_ctype_array_value function a.__ascii_tolower function a.__ascii_toupper function a.__ascii_iswalpha function a.__ascii_iswdigit function a.__ascii_towlower function a.__ascii_towupper function a.__acrt_get_locale_data_prefix function a._chvalidchk_l function a._ischartype_l function a.xa code b function b.__acrt_locale_get_ctype_array_value function b.__ascii_tolower function b.__ascii_toupper function b.__ascii_iswalpha function b.__ascii_iswdigit function b.__ascii_towlower function b.__ascii_towupper function b.__acrt_get_locale_data_prefix function b._chvalidchk_l function b._ischartype_l function b.main C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\bin\HostX64\x64\link.exe /NOLOGO "a.obj" /DEFAULTLIB:phobos64 /LIBPATH:"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.37.32822\lib\x64" legacy_stdio_definitions.lib /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.22621.0\ucrt\x64" /LIBPATH:"C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\um\x64" a.obj : fatal error LNK1179: invalid or corrupt file: duplicate COMDAT '__acrt_locale_get_ctype_array_value' Error: linker exited with status 1179 ------------- This problem does repeat itself for all the functions listed above, if __acrt_locale_get_ctype_array_value is manually mangled appropriately with a hex editor.

The problem stems from when a.c and b.c are compiled together, as in: dmd a.c b.c The code from both gets placed in one object file, a.obj. Hence there are two copies of __acrt_local_get_ctype_array value, and they conflict with each other. The workaround is to compile them separately. The same result is exhibited with this simple test case: --- a.c --- inline int test() { return 73; } void *def() { return &test; } int main() { return 0; } ----- b.c ------- inline int test() { return 73; } void *def() { return &test; } --------------- dmd a.c b.c a.obj : fatal error LNK1179: invalid or corrupt file: duplicate COMDAT 'test' Error: linker exited with status 1179

(In reply to Walter Bright from comment #1) > The problem stems from when a.c and b.c are compiled together, as in: > > dmd a.c b.c > > The code from both gets placed in one object file, a.obj. Hence there are > two copies of __acrt_local_get_ctype_array value, and they conflict with > each other. > > The workaround is to compile them separately. Thanks Walter! Do you think this might be something that can be worked around? The reason is performance, for example, on Linux: $ for i in *.c; do dmd -m64 -fPIC -fPIE -O -release -check=off -boundscheck=off "${i}" -c -of=objout/${i}.o; done $ dmd -m64 -fPIC -fPIE -O -release -check=off -boundscheck=off objout/*.o -of=test1 $ strip test1 $ ls -la test1 885 KB Wed Sep 6 19:29:54 2023 test1* $ rm -rf objout/* $ dmd -m64 -fPIC -fPIE -O -release -check=off -boundscheck=off *.c -of=test2 $ strip test2 $ ls -la test2 838 KB Wed Sep 6 19:30:27 2023 test2* It's quite close, but I can show (with statistical confidence) that the test2 build is a bit faster than test1 (in addition to being smaller and easier to compile). I assume that this is due to some linker optimizations? Does the compiler optimize better, almsot a "poor mans LTO" and treating it as a single translation unit, perhaps? I believe that using -inline (when it works, I made bug #24131 for that) also results in faster binaries when they are all built together, rather than separately. However, as a workaround, this seems acceptable for now, thank you so much. And I assume this is still a bug, so I'll leave this open for now.

It's an important problem to fix. It's a problem only seen with the Microsoft linker - the other linkers automatically remove duplicates, even if they are in the same file. So I'd call it a workaround for an unusual behavior in the Microsoft linker :-) But I can't fix MS-LINK, so I'll just have to remove the duplicates in dmd.

@WalterBright created dlang/dmd pull request #15585 "fix Issue 24129 - ImportC: MS-Link cannot handle multiple COMDATs wit…" fixing this issue: - fix Issue 24129 - ImportC: MS-Link cannot handle multiple COMDATs with the same name https://github.com/dlang/dmd/pull/15585

dlang/dmd pull request #15585 "fix Issue 24129 - ImportC: MS-Link cannot handle multiple COMDATs wit…" was merged into master: - a60429f2eeb738195363ed47f2c519113fe62bd8 by Walter Bright: fix Issue 24129 - ImportC: MS-Link cannot handle multiple COMDATs with the same name https://github.com/dlang/dmd/pull/15585

This is marked RESOLVED FIXED but I get the same error message with DMD 2.108.0.