Bug 21299 – [LINK] undefined reference to dmd.root.stringtable.StringValue!(Type).StringValue.lstring()

Reduced from link error found in regression from Issue 21294 (which caused a refactoring change in dmd to be reverted). It may not be *the* cause of the Issue 21294, but it's certainly a major issue that needs to be addressed, as this is preventing code in the dmd front-end from being moved around. Test script: ``` dmd -c dmd/mtype.d dmd -c dmd/func.d dmd -c dmd/root/stringtable.d dmd -main mtype.o func.o stringtable.o ``` Test sources ``` // dmd/func.d module dmd.func; import dmd.mtype; import dmd.root.stringtable; class FuncDeclaration { StringTable!Type stringtable; } // dmd/mtype.d module dmd.mtype; import dmd.func; import dmd.root.stringtable; class Type { StringTable!Type stringtable; } // dmd/root/stringtable.d module dmd.root.stringtable; struct StringValue(T) { char* lstring() { return cast(char*)&this; } } struct StringTable(T) { StringValue!T* insert() { allocValue; return getValue; } uint allocValue() { StringValue!(T) sv; sv.lstring[0] = 0; return 0; } StringValue!T* getValue() { return cast(StringValue!T*)&this; } } ``` Results in the link error: ``` /usr/bin/ld: mtype.o: in function `_D3dmd4root11stringtable__T11StringTableTCQBo5mtype4TypeZQBe10allocValueMFNaNbNiNfZk': dmd/mtype.d:(.text._D3dmd4root11stringtable__T11StringTableTCQBo5mtype4TypeZQBe10allocValueMFNaNbNiNfZk[_D3dmd4root11stringtable__T11StringTableTCQBo5mtype4TypeZQBe10allocValueMFNaNbNiNfZk]+0x13): undefined reference to `_D3dmd4root11stringtable__T11StringValueTCQBo5mtype4TypeZQBe7lstringMFNaNbNiNfZPa' collect2: error: ld returned 1 exit status Error: linker exited with status 1 ```

What looks to be the reason for the issue: struct StringTable(T) { StringValue!T* insert() fwdrefs -> allocValue; getValue; uint allocValue() instantiates -> StringValue!(T) StringValue!T* getValue() } 1. StringTable!(Type) is having semantic ran from 'dmd.mtype' scope. 2. All members have semantic ran in order of declaration using mtype scope. 3. allocValue is a forward reference inside insert(), so functionSemantic3() is called, which switches to using _scope member, which is the scope of the first instantiation context (not the current) 'dmd.func'. 4. allocValue instantiates StringValue!(Type) using func scope. 5. StringValue!(Type) is appended to the non-root module 'dmd.func', and so not emitted. ------------- Swap the insert() and allocValue() members around though, and the execution path becomes: 1. StringTable!(Type) is having semantic ran from 'dmd.mtype' scope. 2. All members have semantic ran in order of declaration using mtype scope. 3. allocValue instantiates StringValue!(Type) using mtype scope. 4. StringValue!(Type) is appended to the root module 'dmd.mtype', so is emitted. ------------- Immediately then, the problem may be one of: - functionSemantic()/functionSemantic3() doesn't accept a Scope parameter, so the context that they are being evaluated in is lost. They should be fixed to accept a Scope parameter. - The _scope of a function is not updated if a second instantiation occurs in a root module.

In `templateInstanceSemantic`, there is a shortcut for merging multiple template instantiations originating from the same template decl. It fixes up the `minst` Module of the first instantiation to point at the root module instead of a non-root module. However, the first instantiation ran `tryExpandMembers`, which had set the `_scope.minst` of all Dsymbol members to the non-root module. This is left intact despite their parent instance being updated. Why is it important to note this? In the event of handling forward references, all further template instantiations from are pushed into that non-root module that came from `_scope.minst`. It is this that leads to undefined references in both the mtype and func modules, they should have been pushed to the root module, but `_scope.minst` is now incorrectly set. A patch I am preparing for this adds a new Visitor to walk over all members and update their `_scope.minst` to be the root module.

@ibuclaw updated dlang/dmd pull request #11837 "Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring()" fixing this issue: - Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring() In `templateInstanceSemantic`, there exists special handling of matching template instances for the same template declaration to ensure that only at most one instance gets codegen'd. If the primary instance `inst` originated from a non-root module, the `minst` field will be updated so it is now coming from a root module, however all Dsymbol `inst.members` of the instance still have their `_scope.minst` pointing at the original non-root module. We must now propagate `minst` to all members so that forward referenced dependencies that get instantiated will also be appended to the root module, otherwise there will be undefined references at link-time. This doesn't affect compilations where all modules are compiled together, as every module is a root module in that situation. What this primarily affects are cases where there is a mix of root and non-root modules, and a template was first instantiated in a non-root context, then later instantiated again in a root context. https://github.com/dlang/dmd/pull/11837

@ibuclaw created dlang/dmd pull request #11838 "[stable] Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring()" fixing this issue: - Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring() In `templateInstanceSemantic`, there exists special handling of matching template instances for the same template declaration to ensure that only at most one instance gets codegen'd. If the primary instance `inst` originated from a non-root module, the `minst` field will be updated so it is now coming from a root module, however all Dsymbol `inst.members` of the instance still have their `_scope.minst` pointing at the original non-root module. We must now propagate `minst` to all members so that forward referenced dependencies that get instantiated will also be appended to the root module, otherwise there will be undefined references at link-time. This doesn't affect compilations where all modules are compiled together, as every module is a root module in that situation. What this primarily affects are cases where there is a mix of root and non-root modules, and a template was first instantiated in a non-root context, then later instantiated again in a root context. https://github.com/dlang/dmd/pull/11838

dlang/dmd pull request #11838 "[stable] Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring()" was merged into stable: - fb78f0984632f6a9a2cef4dad5f1386115fc978a by Iain Buclaw: Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring() In `templateInstanceSemantic`, there exists special handling of matching template instances for the same template declaration to ensure that only at most one instance gets codegen'd. If the primary instance `inst` originated from a non-root module, the `minst` field will be updated so it is now coming from a root module, however all Dsymbol `inst.members` of the instance still have their `_scope.minst` pointing at the original non-root module. We must now propagate `minst` to all members so that forward referenced dependencies that get instantiated will also be appended to the root module, otherwise there will be undefined references at link-time. This doesn't affect compilations where all modules are compiled together, as every module is a root module in that situation. What this primarily affects are cases where there is a mix of root and non-root modules, and a template was first instantiated in a non-root context, then later instantiated again in a root context. https://github.com/dlang/dmd/pull/11838

dlang/dmd pull request #11867 "[dmd-cxx] Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring()" was merged into dmd-cxx: - 9ab13379acf226069ced667455d834acf217cc66 by Iain Buclaw: [dmd-cxx] Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring() In `templateInstanceSemantic`, there exists special handling of matching template instances for the same template declaration to ensure that only at most one instance gets codegen'd. If the primary instance `inst` originated from a non-root module, the `minst` field will be updated so it is now coming from a root module, however all Dsymbol `inst.members` of the instance still have their `_scope.minst` pointing at the original non-root module. We must now propagate `minst` to all members so that forward referenced dependencies that get instantiated will also be appended to the root module, otherwise there will be undefined references at link-time. This doesn't affect compilations where all modules are compiled together, as every module is a root module in that situation. What this primarily affects are cases where there is a mix of root and non-root modules, and a template was first instantiated in a non-root context, then later instantiated again in a root context. https://github.com/dlang/dmd/pull/11867

dlang/dmd pull request #11874 "Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring()" was merged into stable: - 01afe9cc8f898eafb0b7bd83080ca302e4fbc74c by Iain Buclaw: Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring() Alternative patch where instead of proactively fixing up instance members when a template gets changed to a root instance, rather propagate the `minst` as and when we run semantic on the nested templates that still have a non-root `minst` from its scope. https://github.com/dlang/dmd/pull/11874

dlang/dmd pull request #11886 "merge stable" was merged into master: - 8d6c07f60edfa51e7ad0882def71ef3e6437a465 by Iain Buclaw: Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring() In `templateInstanceSemantic`, there exists special handling of matching template instances for the same template declaration to ensure that only at most one instance gets codegen'd. If the primary instance `inst` originated from a non-root module, the `minst` field will be updated so it is now coming from a root module, however all Dsymbol `inst.members` of the instance still have their `_scope.minst` pointing at the original non-root module. We must now propagate `minst` to all members so that forward referenced dependencies that get instantiated will also be appended to the root module, otherwise there will be undefined references at link-time. This doesn't affect compilations where all modules are compiled together, as every module is a root module in that situation. What this primarily affects are cases where there is a mix of root and non-root modules, and a template was first instantiated in a non-root context, then later instantiated again in a root context. - cf9113d804ec7a89bb354791d66b9555bc4f3f8d by Iain Buclaw: Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring() Alternative patch where instead of proactively fixing up instance members when a template gets changed to a root instance, rather propagate the `minst` as and when we run semantic on the nested templates that still have a non-root `minst` from its scope. https://github.com/dlang/dmd/pull/11886

dlang/dmd pull request #11896 "Revert "Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring()"" was merged into stable: - e2e76d744d3f5f46ecf66ae67cf4089c6711be2e by Iain Buclaw: Revert "Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring()" This reverts commit 32273e82ec9ba3f4022f03e7e12d5d4a384dc3ba. - 74966ae8f8325459df753c3bbe62c54751712ab3 by Iain Buclaw: Issue 21299: Add regression test for another undefined reference issue https://github.com/dlang/dmd/pull/11896

dlang/dmd pull request #11901 "[dmd-cxx] Issue 21299: Add regression test for another undefined reference issue" was merged into dmd-cxx: - 503316612bd7b70bf0e73930710ef27e3494bf03 by Iain Buclaw: [dmd-cxx] Issue 21299: Add regression test for another undefined reference issue https://github.com/dlang/dmd/pull/11901

dlang/dmd pull request #11961 "merge stable" was merged into master: - 92758ee0f100dd2b3bd72714e4db415e1bd876f3 by Iain Buclaw: Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring() In `templateInstanceSemantic`, there exists special handling of matching template instances for the same template declaration to ensure that only at most one instance gets codegen'd. If the primary instance `inst` originated from a non-root module, the `minst` field will be updated so it is now coming from a root module, however all Dsymbol `inst.members` of the instance still have their `_scope.minst` pointing at the original non-root module. We must now propagate `minst` to all members so that forward referenced dependencies that get instantiated will also be appended to the root module, otherwise there will be undefined references at link-time. This doesn't affect compilations where all modules are compiled together, as every module is a root module in that situation. What this primarily affects are cases where there is a mix of root and non-root modules, and a template was first instantiated in a non-root context, then later instantiated again in a root context. - 32273e82ec9ba3f4022f03e7e12d5d4a384dc3ba by Iain Buclaw: Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring() Alternative patch where instead of proactively fixing up instance members when a template gets changed to a root instance, rather propagate the `minst` as and when we run semantic on the nested templates that still have a non-root `minst` from its scope. - 05cbd224e621a858927a1d6e2debbebbb8e246b2 by Iain Buclaw: Revert "Fix Issue 21299: Undefined reference to dmd.root.stringtable.StringValue!(Type).lstring()" This reverts commit 32273e82ec9ba3f4022f03e7e12d5d4a384dc3ba. - cf518229cf08aacee8d681b18b855f94ae8201a6 by Iain Buclaw: Issue 21299: Add regression test for another undefined reference issue https://github.com/dlang/dmd/pull/11961