Bug 20019 – Symbol not found: _dyld_enumerate_tlv_storage on macOS 10.15

Compiling a program using DMD on macOS Catalina (10.15) beta 2 gives the following error: dyld: lazy symbol binding failed: Symbol not found: _dyld_enumerate_tlv_storage Referenced from: ~/.dvm/compilers/dmd-2.086.0/osx/bin/dmd Expected in: /usr/lib/libSystem.B.dylib dyld: Symbol not found: _dyld_enumerate_tlv_storage Referenced from: ~/.dvm/compilers/dmd-2.086.0/osx/bin/dmd Expected in: /usr/lib/libSystem.B.dylib Abort trap: 6 "dyld_enumerate_tlv_storage" was a private symbol part of dyld (the dynamic loader). It is used to get the memory locations of all the thread local storage variables to register them as roots with the garbage collector. This symbol doesn't seem to exist anymore on macOS Catalina. This issue will apply to LDC as well.

Actually, just running DMD is enough to reproduce it.

> This issue will apply to LDC as well. Nope, we don't use this symbol.

Argh, my grep was bad, as the C druntime files aren't included in a prebuilt package (I'm at work), so yes, LDC will be affected too.

@jacob-carlborg created dlang/druntime pull request #2666 " Fix issue 20019: Symbol not found: _dyld_enumerate_tlv_storage on macOS 10.15" fixing this issue: - Fix issue 20019: Symbol not found: _dyld_enumerate_tlv_storage on macOS 10.15 The symbol `dyld_enumerate_tlv_storage` has been used to iterate all the thread local storage addresses and registered them as roots with the garbage collector. The symbol has always been a private symbol, not part of the public API. In the latest version of macOS, 10.15 it has been removed. This replaces the usage of `dyld_enumerate_tlv_storage` by manually iterating the TLV sections of all images and get key that the dynamic loader has setup for each TLS symbol. The key is then used to get the base address of the TLV storage. https://github.com/dlang/druntime/pull/2666

dlang/druntime pull request #2666 " Fix issue 20019: Symbol not found: _dyld_enumerate_tlv_storage on macOS 10.15" was merged into stable: - 378fc6dfadc8b638616d146496eaee79995678ac by Jacob Carlborg: Fix issue 20019: Symbol not found: _dyld_enumerate_tlv_storage on macOS 10.15 The symbol `dyld_enumerate_tlv_storage` has been used to iterate all the thread local storage addresses and registered them as roots with the garbage collector. The symbol has always been a private symbol, not part of the public API. In the latest version of macOS, 10.15 it has been removed. This replaces the usage of `dyld_enumerate_tlv_storage` by manually iterating the TLV sections of all images and get key that the dynamic loader has setup for each TLS symbol. The key is then used to get the base address of the TLV storage. https://github.com/dlang/druntime/pull/2666

I seem to have gotten in an unsolvable situation with reinstalling due to (it seems) this issue on an osx machine. 1. First I cleaned everything there is and then synced everything with git. 2. In the dmd directory I issued: $ make -f posix.mak AUTO_BOOTSTRAP=1 To my surprise that did not actually build the compiler via bootstrapping, it just installed a prebuilt binary 2.088.0. Why did we change that? Bootstrap should bootstrap. 3. Then I tried in the same dmd directory the command: $ make -f posix.mak And the first line executed in the makefile was: dmd -of../generated/build -g build.d ...which failed with: Error: cannot find source code for runtime library file 'object.d' dmd might not be correctly installed. Run 'dmd -man' for installation instructions. config file: /Users/andrei/bin/dmd.conf A few questions. A. Why doesn't "AUTO_BOOTSTRAP=1" bootstrap anymore? B. Even if it doesn't go through all stages of bootstrapping, why doesn't it leave a working installation behind? My /Users/andrei/bin/dmd.conf completely messed that up. It should warn me about its existence or overwrite it leaving a backup behind. c. Why is the build process looking at /Users/andrei/bin/dmd.conf? It should be self-contained, i.e. not assume there's any working installation and override all possible user-defined flags and configuration files.

Update: I moved that conf file out of the way, things still don't work. After running with AUTO_BOOTSTRAP=1, I assume I had a good installation. But: $ cd /path/to/dmd $ make -f posix.mak dmd -of../generated/build -g build.d Error: cannot find source code for runtime library file 'object.d' dmd might not be correctly installed. Run 'dmd -man' for installation instructions. config file: /Users/andrei/d/dmd/generated/osx/release/64/dmd.conf Why didn't AUTO_BOOTSTRAP leave behind a working installation?

(In reply to Andrei Alexandrescu from comment #6) > [...] > To my surprise that did not actually build the compiler via bootstrapping, > it just installed a prebuilt binary 2.088.0. Why did we change that? > Bootstrap should bootstrap. The name is quite misleading, something else might have been more appropriate. But this behaviour hasn't changed (except minor improvements) since you introduced it some years ago (see https://github.com/dlang/dmd/commit/a1217f92f596bd9ce07f7c327a6980797085ba7d). > make -f posix.mak AUTO_BOOTSTRAP=1 This compiles dmd using the fetched dmd installed to generated/host_dmd-2.088.0 as expected. Did some error occur on your end? > 3. Then I tried in the same dmd directory the command: > > $ make -f posix.mak This doesn't work because the Makefile does not know about the "bootstrapped" compiler and tries to use dmd found in $PATH. Might be worth to add some logic to detect this. > Why didn't AUTO_BOOTSTRAP leave behind a working installation? Did you build druntime & phobos as well?

(In reply to moonlightsentinel from comment #8) > Did you build druntime & phobos as well? Please ignore this. Did you set some environment variables used in the makefile aside from AUTO_UPDATE?

> Did you set some environment variables used in the > makefile aside from AUTO_UPDATE? I don't have any dmd-specific envvars that I know of. Did you mean s/AUTO_UPDATE/AUTO_BOOTSTRAP/? (I didn't use AUTO_UPDATE and don't know what it is.)

I recommend to manually download the latest version of DMD and use that. The fix for this issue was included in version 2.087.1. DMD is always built with the previous version. If the bootstrap downloads DMD 2.088.0, I don't know if that is built using 2.087.0 or 2.087.1. If it's built using 2.087.0 then this fix is missing.

(In reply to Andrei Alexandrescu from comment #10) > > Did you set some environment variables used in the > > makefile aside from AUTO_UPDATE? > > I don't have any dmd-specific envvars that I know of. Did you mean > s/AUTO_UPDATE/AUTO_BOOTSTRAP/? (I didn't use AUTO_UPDATE and don't know what > it is.) Yes, i meant AUTO_BOOTSTRAP. (AUTO_UPDATE is used in dmds test suite)

(In reply to Jacob Carlborg from comment #11) > I recommend to manually download the latest version of DMD and use that. > > The fix for this issue was included in version 2.087.1. DMD is always built > with the previous version. If the bootstrap downloads DMD 2.088.0, I don't > know if that is built using 2.087.0 or 2.087.1. If it's built using 2.087.0 > then this fix is missing. Might be worth a try to change the compiler version specified in posix.mak (https://github.com/dlang/dmd/blob/91107b4750a82f8b86aa87e9155bed3216c68106/src/posix.mak#L91) and check whether it still fails with the current version.