When using std.net.curl over SSL the underlying OpenSSL implementation isn't thread safe unless some mutex callbacks are set.
This leads to rece conditions hen parallely downloading multiple files.
See curl docs for a thread-safe example.
http://curl.haxx.se/libcurl/c/opensslthreadlock.html
Comment #1 by brocolis — 2015-07-20T01:15:24Z
Just in case someone else get hit by this bug, here's a working example for linux.
--
import std.net.curl;
import std.parallelism;
import std.range;
import std.stdio;
void main()
{
init_locks();
foreach(i; parallel(iota(0, 10, 1)))
{
writefln("Testing %d ...", i);
get("https://...");
}
kill_locks();
}
import core.sync.mutex;
import core.stdc.config;
import core.sys.posix.pthread;
alias extern(C) c_ulong function() CRYPTO_set_id_callback_t;
alias extern(C) void function(int, int, const(char)*, int) CRYPTO_set_locking_callback_t;
extern(C) int CRYPTO_num_locks();
extern(C) void CRYPTO_set_id_callback(CRYPTO_set_id_callback_t);
extern(C) void CRYPTO_set_locking_callback(CRYPTO_set_locking_callback_t);
enum CRYPTO_LOCK = 1;
__gshared Mutex[] mutex;
void init_locks()
{
mutex.length = CRYPTO_num_locks();
for (int i = 0; i < mutex.length; i++)
{
mutex[i] = new Mutex;
}
CRYPTO_set_id_callback(&thread_id);
CRYPTO_set_locking_callback(&lock_callback);
}
void kill_locks()
{
CRYPTO_set_id_callback(null);
CRYPTO_set_locking_callback(null);
foreach(m; mutex)
destroy(m);
mutex.length = 0;
}
extern(C) void lock_callback(int mode, int n, const(char)* file, int line)
{
if (mode & CRYPTO_LOCK)
{
mutex[n].lock();
}
else
{
mutex[n].unlock();
}
}
extern(C) c_ulong thread_id()
{
return pthread_self();
}
--
Comment #2 by bugzilla — 2019-12-17T08:51:48Z
Getting linker errors, when trying the program von brocolis. Which compiler flags do I need to get access to CRYPTO_num_locks etc.?
Comment #3 by brocolis — 2019-12-17T09:59:27Z
dmd bug.d -L-lssl
Comment #4 by bugzilla — 2019-12-17T12:31:28Z
I'm still getting those linker errors. The same, when I use run.dlang.io
Comment #5 by brocolis — 2019-12-21T18:09:59Z
'dmd bug.d -L-lssl' just works for me (unmodified code from 2015-07)
Comment #6 by robert.schadek — 2024-12-01T16:19:27Z