Bug 15391 – Problems loading libcurl.so and running datetime unittest on NixOS package build

I tried to run the unittests of phobos 2.069.1 but got the following deadlock in test_runner while running the std.net.curl unittest: (gdb) thread apply all bt Thread 2 (Thread 0x7ffff1ce5700 (LWP 19261)): #0 0x00007ffff27a47dd in accept () from target:/nix/store/npfsi1d9ka8zwnxzn3sr08hbwvpapyk7-glibc-2.21/lib/libpthread.so.0 #1 0x00007ffff624eae4 in std.socket.Socket.accept() (this=0x7ffff7ef1700) at std/socket.d:2855 #2 0x00007ffff6f7fb8a in std.net.curl.TestServer.loop(shared(std.socket.TcpSocket)) ( listener=0x7ffff7ef1700) at std/net/curl.d:213 #3 0x00007ffff6fad46d in std.concurrency._spawn!(void(shared(std.socket.TcpSocket)) function*, shared(std.socket.TcpSocket))._spawn(bool, void(shared(std.socket.TcpSocket)) function*, shared(std.socket.TcpSocket)).exec() (this=0x7ffff7ef2100) at std/concurrency.d:517 #4 0x00007ffff75c903f in core.thread.Thread.run() (this=0x7ffff7eee500) at src/core/thread.d:1351 #5 0x00007ffff75c7f6c in thread_entryPoint (arg=0x60d850) at src/core/thread.d:347 #6 0x00007ffff279d484 in start_thread () from target:/nix/store/npfsi1d9ka8zwnxzn3sr08hbwvpapyk7-glibc-2.21/lib/libpthread.so.0 #7 0x00007ffff1dcd05d in clone () from target:/nix/store/npfsi1d9ka8zwnxzn3sr08hbwvpapyk7-glibc-2.21/lib/libc.so.6 Thread 1 (Thread 0x7ffff7ff25c0 (LWP 19260)): #0 0x00007ffff279e65d in pthread_join () from target:/nix/store/npfsi1d9ka8zwnxzn3sr08hbwvpapyk7-glibc-2.21/lib/libpthread.so.0 #1 0x00007ffff75c85b5 in core.thread.Thread.join(bool) (this=0x7ffff7eee500, rethrow=true) at src/core/thread.d:741 #2 0x00007ffff75c9b7f in thread_joinAll () at src/core/thread.d:2283 #3 0x00007ffff75eed0b in rt_term () at src/rt/dmain2.d:207 #4 0x00007ffff75ef240 in rt.dmain2._d_run_main(int, char**, extern(C) int(char[][]) function*).runAll() (this=0x7fffffffd990) at src/rt/dmain2.d:476 #5 0x00007ffff75ef1c1 in rt.dmain2._d_run_main(int, char**, extern(C) int(char[][]) function*).tryExec(scope void() delegate) (this=0x7fffffffd990, dg=...) at src/rt/dmain2.d:447 #6 0x00007ffff75ef12b in _d_run_main (argc=2, argv=0x7fffffffda98, mainFunc=0x4017b0 <D main>) at src/rt/dmain2.d:480 #7 0x0000000000401988 in main () #8 0x00007ffff1d06995 in __libc_start_main () from target:/nix/store/npfsi1d9ka8zwnxzn3sr08hbwvpapyk7-glibc-2.21/lib/libc.so.6 #9 0x00000000004012e9 in _start ()

I need to add that I try to run the unittests in the checkPhase of a package build of NixOS and the network is restricted in this case I guess. But the tests for std.socket were successful: 0.518s PASS debug64 std.socket

Running the testsuite in the shell instead of inside the package creation process gives me an Exception and the process deadlocks in the same place. So the libcurl dependency cannot be resolved. ****** FAIL debug64 std.net.curl std.net.curl.CurlException@std/net/curl.d(3885): Failed to load curl, tried "libcurl.so", "libcurl.so.4", "libcurl-gnutls.so.4", "libcurl-nss.so.4", "libcurl.so.3".

I was able to hack around the issue by patching the curl.d sourcefile. #Ugly hack so the dlopen call has a chance to succeed. substituteInPlace src/phobos/std/net/curl.d --replace libcurl.so ${curl}/lib/libcurl.so That's in the package description for the dmd package on NixOS. That's necessary because in NixOS every package is stored in isolation to everything else. So libcurl on my system resides in /nix/store/v5a69m1b823zm1yy8yvhrd2zi2w385b1-curl-7.44.0/lib/libcurl.so. This also makes problems in the unittest of datetime.d. The path to the zoneinfo is hardcoded so I needed to hack around it in a similar fashion. #Ugly hack to fix the hardcoded path to zoneinfo in the source file. substituteInPlace src/phobos/std/datetime.d --replace /usr/share/zoneinfo/ ${tzdata}/share/zoneinfo/ I think it should be possible to specify those paths somehow via the Makefile to get a clean build.

What are you doing differently from normal that makes it so that the normal path for the time zone files doesn't work?

(In reply to Thomas Mader from comment #3) > I was able to hack around the issue by patching the curl.d sourcefile. > > #Ugly hack so the dlopen call has a chance to succeed. > substituteInPlace src/phobos/std/net/curl.d --replace libcurl.so > ${curl}/lib/libcurl.so > > > That's in the package description for the dmd package on NixOS. That's > necessary because in NixOS every package is stored in isolation to > everything else. > So libcurl on my system resides in > /nix/store/v5a69m1b823zm1yy8yvhrd2zi2w385b1-curl-7.44.0/lib/libcurl.so. Maybe try to LD_LIBRARY_PATH instead? As mentioned here https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading you can also link against libcurl (make sure to use no-as-needed), then the symbols will get loaded from your executable.

(In reply to Jonathan M Davis from comment #4) > What are you doing differently from normal that makes it so that the normal > path for the time zone files doesn't work? NixOS is using a new and different approach to package and system management as normal Linux distributions do and one aspect of that is that there is no such thing as standard paths. In my next reply I will attach a link to a better explanation about the reasoning behind the NixOS ideas.

(In reply to Martin Nowak from comment #5) > (In reply to Thomas Mader from comment #3) > > I was able to hack around the issue by patching the curl.d sourcefile. > > > > #Ugly hack so the dlopen call has a chance to succeed. > > substituteInPlace src/phobos/std/net/curl.d --replace libcurl.so > > ${curl}/lib/libcurl.so > > > > > > That's in the package description for the dmd package on NixOS. That's > > necessary because in NixOS every package is stored in isolation to > > everything else. > > So libcurl on my system resides in > > /nix/store/v5a69m1b823zm1yy8yvhrd2zi2w385b1-curl-7.44.0/lib/libcurl.so. > > Maybe try to LD_LIBRARY_PATH instead? > As mentioned here > https://dlang.org/changelog/2.069.0.html#curl-dynamic-loading you can also > link against libcurl (make sure to use no-as-needed), then the symbols will > get loaded from your executable. The LD_LIBRARY_PATH solution isn't viable for NixOS and even if the libcurl linking is solved, two issues remain. (Path to zoneinfo and using gcc's linker) I think I need to explain the idea of NixOS a little bit. Since I am new to NixOS I am not the perfect candidate to explain in detail. Therefore it's better to read https://www.domenkozar.com/2014/03/11/why-puppet-chef-ansible-arent-good-enough-and-we-can-do-better/ for a brief overview and the reasoning behind the NixOS idea. I see the patching I need to do now to package dmd on NixOS as a Hack. There needs to be a better solution. Wouldn't it be reasonable to provide a way to define the exact path to every external dependency? I thought about optional Makefile arguments to set the paths which are propageted into the build as compiler switches? If it is considered a viable solution, I could even try to do a PR.

(In reply to Thomas Mader from comment #3) > I was able to hack around the issue by patching the curl.d sourcefile. > > #Ugly hack so the dlopen call has a chance to succeed. > substituteInPlace src/phobos/std/net/curl.d --replace libcurl.so > ${curl}/lib/libcurl.so > > > That's in the package description for the dmd package on NixOS. That's > necessary because in NixOS every package is stored in isolation to > everything else. > So libcurl on my system resides in > /nix/store/v5a69m1b823zm1yy8yvhrd2zi2w385b1-curl-7.44.0/lib/libcurl.so. With dmd 2.075.0 I also needed to patch the Makefile of phobos to include the full path to the shared lib: substituteInPlace phobos/posix.mak \ --replace "-soname=libcurl.so.4" "-soname=${curl.out}/lib/libcurl.so.4"

Commit pushed to master at https://github.com/dlang/phobos https://github.com/dlang/phobos/commit/735b360c4215720adf3b64b76c274ce1094538f9 Fix Issue 15391 - Add possibility to specify custom path to TZDatabaseDir