Bug 15060 – Can't load a D shared library first, then load a C shared library

Repost of a message in https://issues.dlang.org/show_bug.cgi?id=14824 also reported here with a $50 bounty since I originally found it with LDC, but it also happens with DMD: https://github.com/ldc-developers/ldc/issues/1071 DMD version = 2.068.1 OS = OS X 10.10.4 == Setup == Here the host program source. What it does is: - for each dynlib in command line: - load dynlibs - call the VSTPluginMain function if it exist, - unload it --------------------- ldvst.cpp -------------------- #include <dlfcn.h> #include <cstdio> #include <cstring> #include <vector> typedef __cdecl void* (*VSTPluginMain_t)(void*); int main(int argc, char**argv) { std::vector<char*> dllPaths; if (argc < 2) { printf("usage: ldvst [-lazy] <thing.vst>\n"); return 1; } bool lazy = false; for (int i = 1; i < argc; ++i) { char* arg = argv[i]; if (strcmp(arg, "-lazy") == 0) lazy = true; else if (strcmp(arg, "-now") == 0) lazy = false; else dllPaths.push_back(arg); } for (int i = 0; i < dllPaths.size(); ++i) { char* dllPath = dllPaths[i]; printf("dlopen(%s)\n", dllPath); void* handle = dlopen(dllPath, lazy ? RTLD_LAZY : RTLD_NOW); if (handle == NULL) { printf("error: dlopen of %s failed\n", dllPath); return 2; } VSTPluginMain_t VSTPluginMain = (VSTPluginMain_t) dlsym(handle, "VSTPluginMain"); printf("dlsym returned %p\n", (void*)VSTPluginMain); if (VSTPluginMain != NULL) { void* result = VSTPluginMain(NULL); printf("VSTPluginMain returned %p\n", result); } printf("dlclose(%s)\n\n", dllPath); dlclose(handle); } return 0; } ----------------------------------------------------- The host is compiled with: $ clang++ ldvst.cpp -o ldvst-cpp Here the whole dynlib source: ----------------- distort.d ------------------------- extern(C) void* VSTPluginMain(void* hostCallback) { import core.runtime; Runtime.initialize(); import std.stdio; import core.stdc.stdio; printf("Hello !\n"); Runtime.terminate(); return null; } ------------------------------------------------------ This dynlib can be built with ldc: $ ldc2 -shared -oflibdistort.so -g -w -I. distort.d -relocation-model=pic or with dmd $ dmd -c -ofdistort.o -debug -g -w -version=Have_distort -I. distort.d -fPIC $ dmd -oflibdistort.so distort.o -shared -g For the purpose of demonstration you need another C dynlib, for example /System/Library/Frameworks/Cocoa.framework/Cocoa on OS X. Now the bug triggers when calling: == How to reproduce == $ ldvst-cpp libdistort.so => works $ ldvst-cpp libdistort.so libdistort.so => works $ ldvst-cpp /System/Library/Frameworks/Cocoa.framework/Cocoa libdistort.so => works $ ldvst-cpp libdistort.so /System/Library/Frameworks/Cocoa.framework/Cocoa => FAIL, and that's precisely the case that happen in production :( $ ldvst-cpp /System/Library/Frameworks/Cocoa.framework/Cocoa libdistort.so /System/Library/Frameworks/Cocoa.framework/Cocoa => works In other words, if the host program loads a D dynlib first, then a C dynlib, then the second dlopen fail. This is pernicious since the host program would scan my program, dlclose successfully, then put in jail another program. As of today, this is the only thing between me and customers, all other bugs can be work-arounded, this one I'm not sure.

Suggestion from David Nadlinger " This is OS X only, right? In this case, it is hard to judge where exactly it sits and what needs to be done to fix it, as no work has been done on dylib support recently in either compiler. A wild guess is that there might still be an image change handler, etc. registered that dyld tries to call when loading the new library, but which no longer exists in the program. Searching druntime for "_dyld" might reveal something of this sort."

I found a 2012 conversation here that talks about this very problem: https://github.com/D-Programming-Language/druntime/pull/228

Hacky workaround found by bitwise and Martin Nowak: http://forum.dlang.org/post/[email protected]

Can someone please try to fix this properly in druntime? If it's not doable using the dyld API, I might work on adding init/fini calls to any OSX binary (similar to what we do on ELF).

Unfortunately I'm trying to bootstrap a business so I'll live with the hacky work-around and move on to the next task (I do everything). It's also way beyond my understanding. I've bountied $50 on the equivalent bug in LDC, this isn't much, so if someone want to do this, quote your price here (I would move the bounty on the proper bug but not sure how to do it). https://github.com/ldc-developers/ldc/issues/1071 https://www.bountysource.com/issues/26587024-can-t-load-a-d-shared-library-first-then-load-a-c-shared-library

For reference: dyld's source: http://www.opensource.apple.com/source/dyld/ http://www.opensource.apple.com/source/dyld/dyld-353.2.3/ The callbacks being discussed: '_dyld_register_func_for_add_image' '_dyld_register_func_for_remove_image' https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/dyld.3.html https://developer.apple.com/library/mac/documentation/DeveloperTools/Reference/MachOReference/index.html 'dyld_register_image_state_change_handler' couldn't find docs, but declaration/definition are here: http://www.opensource.apple.com/source/dyld/dyld-353.2.3/include/mach-o/dyld_priv.h http://www.opensource.apple.com/source/dyld/dyld-353.2.3/src/dyldAPIs.cpp Finally, I don't see any callback API which allows you to unregister the "library loaded" callback, so I think that adding init/fini calls is the solution. Now, when the library is loaded, and init/fini are called, we need a way to get the mach_header* for the library which is loading. 'dladdr' can do this. It's documented that the 'dli_fbase' member of dl_info is a (mach_header*). So a private symbol can be added to the files along with the init/fini calls...or I suppose you could use the address of the init/fini calls themselves.. and pass the symbol to dladdr. https://developer.apple.com/library/mac/documentation/DeveloperTools/Reference/MachOReference/index.html#//apple_ref/c/func/dladdr The other piece of information that you need is the vmaddr_slide value. Right now, '_dyld_get_image_slide' is listed in 'dyld_priv.h'. If it turns out that _dyld_get_image_slide is a private API or something, and not allowed in the Apple Store, we can use plan B, which is to just iterate the mach_images and vmaddr_slides in lockstep until a match is found using these APIs: uint32_t _dyld_image_count() const struct mach_header* _dyld_get_image_header(uint32_t image_index) intptr_t _dyld_get_image_vmaddr_slide(uint32_t image_index) These three are APIs are here: https://developer.apple.com/library/mac/documentation/DeveloperTools/Reference/MachOReference/index.html '_dyld_get_image_slide' can be found here: http://www.opensource.apple.com/source/dyld/dyld-353.2.3/include/mach-o/dyld_priv.h Last time I looked, it seemed like this is more complicated than what you've done for ELF. I think manually injecting this code in the backend as ASM would be very difficult. I would(if I knew how) instead try to add some pragmas to DMD to help out, so that init/fini could just be written in D. Then, DMD's front end could inject the needed code. For example, LDC already has these, so maybe DMD could too: pragma(LDC_global_crt_ctor, 1024) pragma(LDC_global_crt_dtor, 1024) http://wiki.dlang.org/LDC-specific_language_changes I think one more could make symbols private(GCC has something like this) pragma(visibility, hidden) Since public symbols are merged even between shared libraries on OSX, init/fini and whatever symbol is passed to dladdr would have to be private/hidden. Maybe one final pragma to make the symbols COMDAT.

(In reply to Martin Nowak from comment #4) > Can someone please try to fix this properly in druntime? > If it's not doable using the dyld API, I might work on adding init/fini > calls to any OSX binary (similar to what we do on ELF). I've looked in the source code of dyld. There's no code that removes anything from the list of callbacks. I don't see any other way than using the init/fini calls.

>> ponce 2015-09-18 09:47:37 UTC >> Hacky workaround found by bitwise and Martin Nowak: >> http://forum.dlang.org/post/[email protected] I suppose I should note that I was expanding on Jacob's solution ;)

If this need back-end support, maybe Walter could implement global ctor/dtor like in LDC? @bitwise: how much would you take to do it? :)) FWIW, I'm having problem with the workaround (calling dyld_register_image_state_change_handler crash). It's as if dyld_register_image_state_change_handler has a different signature or calling convention in 32-bit.

(In reply to ponce from comment #9) > If this need back-end support, maybe Walter could implement global ctor/dtor > like in LDC? DMD already does what's necessary for ELF (Linux, FreeBSD). Just do the same for Mach-O and OS X. > FWIW, I'm having problem with the workaround (calling > dyld_register_image_state_change_handler crash). It's as if > dyld_register_image_state_change_handler has a different signature or > calling convention in 32-bit. The declaration of dyld_register_image_state_change_handler is available here [1]. The declaration of "dyld_image_info" [2] and "mach_header" [3]. Looking at your implementation, both the function pointer for the callback and the function it points to need to be declared as extern(C). [1] http://www.opensource.apple.com/source/dyld/dyld-96.2/include/mach-o/dyld_priv.h [2] http://opensource.apple.com/source/dyld/dyld-132.13/include/mach-o/dyld_images.h [3] http://www.opensource.apple.com/source/xnu/xnu-1456.1.26/EXTERNAL_HEADERS/mach-o/loader.h

:X Thanks!!! can't believe I forgot that.

(In reply to Jacob Carlborg from comment #10) > (In reply to ponce from comment #9) > > If this need back-end support, maybe Walter could implement global ctor/dtor > > like in LDC? > > DMD already does what's necessary for ELF (Linux, FreeBSD). Just do the same > for Mach-O and OS X. >> DMD already does what's necessary for ELF (Linux, FreeBSD). Just do the same >> for Mach-O and OS X. That's easier said than done ;) Making druntime compile as a shared lib and making it support multiple images is the easy part. Modifying DMD to output the ctors/dtors properly is much more difficult than it sounds. The current solution outputs the ctors/dtors by manually outputting byte codes: https://github.com/D-Programming-Language/dmd/blob/master/src/backend/elfobj.c#L3183 This is why I was saying we should just add some(or one special one) pragmas that will allow this to be done in the front end. >> @bitwise: how much would you take to do it? :)) Wish I knew how =/

(In reply to bitwise from comment #12) > (In reply to Jacob Carlborg from comment #10) > >> DMD already does what's necessary for ELF (Linux, FreeBSD). Just do the same > >> for Mach-O and OS X. > > That's easier said than done ;) :) > This is why I was saying we should just add some(or one special one) pragmas > that will allow this to be done in the front end. I don't think this can be completely done in the front end. The code (or that function) that should be run as a global ctor/dtor need to be placed in a special segment/section, if I recall correctly.

(In reply to Jacob Carlborg from comment #13) > (In reply to bitwise from comment #12) > > (In reply to Jacob Carlborg from comment #10) > > >> DMD already does what's necessary for ELF (Linux, FreeBSD). Just do the same > > >> for Mach-O and OS X. > > > > That's easier said than done ;) > > :) > > > This is why I was saying we should just add some(or one special one) pragmas > > that will allow this to be done in the front end. > > I don't think this can be completely done in the front end. The code (or > that function) that should be run as a global ctor/dtor need to be placed in > a special segment/section, if I recall correctly. Right, which is why I am recommending the special pragmas(or pragma).

(In reply to bitwise from comment #14) > (In reply to Jacob Carlborg from comment #13) > > [...] > > Right, which is why I am recommending the special pragmas(or pragma). Basically, we need these: pragma(attribute, weak) pragma(visibility, hidden) pragma(section, "__mod_init_func"); pragma(section, "__mod_term_func");

THIS ISSUE HAS BEEN MOVED TO GITHUB https://github.com/dlang/dmd/issues/17146 DO NOT COMMENT HERE ANYMORE, NOBODY WILL SEE IT, THIS ISSUE HAS BEEN MOVED TO GITHUB