Bug 9737 – std.net.curl and SSL - documentation missing

I wanted to use std.net.curl to make a simple request over SSL, for ex. get("https://google.com"). It failed with the message "Peer certificate cannot be authenticated with given CA certificates". In documentation I could not find anything, so it was a tedious search for something that has actually an easy resolution. Since this is a quite common usage scenario, it should be documented. Also note that the HTTP.caInfo does not have any doc entry. SSL usage example: ----- auto http = HTTP(); // Set the CA certificate bundle file to use for SSL peer verification // To dowload this file (or generate it yourself) see http://curl.haxx.se/docs/caextract.html // For more information about SSL peer verification see http://curl.haxx.se/docs/sslcerts.html http.caInfo("cacert.pem"); //http.handle.set(CurlOption.ssl_verifypeer, 0); // do not verify ssl certifiactes, use with care! auto resp = get("https://google.com", http); -----

Commits pushed to master at https://github.com/D-Programming-Language/phobos https://github.com/D-Programming-Language/phobos/commit/cab6c5837934468e9eb67bfc498f13a1d2088e10 Fix Issue 9737 - std.net.curl and SSL - documentation missing https://github.com/D-Programming-Language/phobos/commit/33114ec2ea612e19d7b3c251c5daeea5e617dd93 Merge pull request #1687 from brocolis/fix9737 Fix Issue 9737 - std.net.curl and SSL - documentation missing

Documentation should help the user understand something they _don't already know about_. This doesn't offer anything to me as a user wondering why curl fails on some URL's. There's nothing even in the error message that suggests that it's SSL/HTTPS related. Encryption is complex, one sentence that doesn't say anything is not going to help new users (like me).

Feel free to reopen and make another pull that adds the appropriate docs. Thanks.

As someone who actually wanted to read this documentation, and knows almost nothing about HTTPS, I can't possibly be considered an authority on the topic. I just suggest that this bug can't be closed. Since as an end user wanting to read this article, I'm left with exactly no more understanding with this sentence than I had without it. The only value I gained is a search term I can punch into Google, which I argue isn't a very good form of documentation. This bug should remain open until someone who knows what they're talking about writes some decent documentation. There's no hurry to close it.

I agree that the line of documentation added is not a 'how does https' work, or even a 'what does libcurl need to have done to make https work'. However, it is a reasonable description of what that api does. This issue boils down, I believe, to be essentially, libcurl was incompletely setup on the reporters box and a faq entry would be handy. A properly installed libcurl includes the cert bundle that libcurl uses automatically by default, allows https urls to 'just work'. For non-windows platforms, this is done by the platforms' packaging of libcurl. For the windows platform, it's libcurl doesn't have one by default and the urls in the original report address the situation pretty well. So, some additional documentation near the top of the std.net.curl (and etc.c.curl) would be useful. A few pointers off to the those curl doc pages, maybe one or two good https background docs. However, I do not believe that inside the phobos docs is a reasonable place to put documentation about the general topic of how https works and even more generally how encryption works.

(In reply to comment #5) > I agree that the line of documentation added is not a 'how does https' work, or > even a 'what does libcurl need to have done to make https work'. However, it > is a reasonable description of what that api does. But this bug doesn't say "document this API", it says "std.net.curl and SSL - documentation missing", which is still true, so this bug is still open for my money. > This issue boils down, I believe, to be essentially, libcurl was incompletely > setup on the reporters box and a faq entry would be handy. A properly > installed libcurl includes the cert bundle that libcurl uses automatically by > default, allows https urls to 'just work'. For non-windows platforms, this is > done by the platforms' packaging of libcurl. For the windows platform, it's > libcurl doesn't have one by default and the urls in the original report address > the situation pretty well. > > So, some additional documentation near the top of the std.net.curl (and > etc.c.curl) would be useful. A few pointers off to the those curl doc pages, > maybe one or two good https background docs. However, I do not believe that > inside the phobos docs is a reasonable place to put documentation about the > general topic of how https works and even more generally how encryption works. I think an important point is, I didn't even realise that *HTTPS* specifically didn't work, I was just annoyed that a seemingly trivial API [get(url)] didn't just work. After I worked out that it was HTTPS related, there was no mention of HTTPS anywhere in the docs, or examples, or that it required special treatment. What you say is fine, it's not the place for an intro to SSL/encryption, but it needs to be made clear that HTTPS is not trivial, and requires special handling, and the error message needs to be improved so programmer's aren't left wondering why it didn't work. That said, I think there's a secondary 'bug' here, that is, std.net.curl is broken on some platforms out of the box. We're talking about a lib under 'std' here. If it's part of phobos, then it shouldn't depend on anything that's not distributed with phobos. Consider any other modern language, python, java, C#, PHP, Ruby... if I make a HTTP request, it works. I think that's a reasonable end-user expectation. Basically all websites are HTTPS these days, it needs to just work too. If it requires ancilliary files, I'd say it's the DMD/GDC/LDC distributions responsibility to make sure these api's do just work. Documenting that it doesn't work (but you can make it work by following these simple steps) is isn't acceptable. Why isn't the file which just seems to be full of (standard?) magic keys embedded in curl or phobos?

(In reply to comment #6) > Consider any other modern language, python, java, C#, PHP, Ruby... if I make a > HTTP request, it works. I think that's a reasonable end-user expectation. > Basically all websites are HTTPS these days, it needs to just work too. We had some problems with SSL certificates when we moved from Ruby 1.8 to 1.9. I don't remember the exact error but it didn't work out of the box.

I see the certificate bundle file (and the scripts to regenerate it) are installed by dmd in the ./dmd2/windows folder It seems that the file is not picked automatically by libcurl even if I copy it along the EXE (tried with ca-bundle.crt and curl-ca-bundle.crt). So on Windows would be optimal if std.net.curl would set automatically the option, if the ca-bundle.crt is in the same folder as the EXE (or make this configurable). Then the documentation should specify that you need to deliver with your EXE the dlang/dmd2/windows/ca-bundle.crt

(In reply to comment #8) > I see the certificate bundle file (and the scripts to regenerate it) are > installed by dmd in the ./dmd2/windows folder > It seems that the file is not picked automatically by libcurl even if I copy it > along the EXE (tried with ca-bundle.crt and curl-ca-bundle.crt). So on Windows > would be optimal if std.net.curl would set automatically the option, if the > ca-bundle.crt is in the same folder as the EXE (or make this configurable). > Then the documentation should specify that you need to deliver with your EXE > the dlang/dmd2/windows/ca-bundle.crt Can't we build one in, into Phobos using import("ca-bundle.crt");?

THIS ISSUE HAS BEEN MOVED TO GITHUB https://github.com/dlang/phobos/issues/9963 DO NOT COMMENT HERE ANYMORE, NOBODY WILL SEE IT, THIS ISSUE HAS BEEN MOVED TO GITHUB