Bug 16349 – better curl retry for install.sh script

Sometimes the install script fails b/c of some temporary network failure. All downloads should use some limited retry mechanism to recover from those. This problem is particularly annoying since it leads to spurious Travis-CI failures.

Oh sorry I didn't know this was already a known issue before I reported it at Travis (https://github.com/travis-ci/travis-ci/issues/6401). It's definitely good if the installation gets more robust, but maybe we can also dig into the root cause of the random network failures. I never experienced such constant problems with other providers, do you know anything what could cause this? Should we consider to provide a more stable distribution way like e.g. Github Releases?

(In reply to greenify from comment #1) > It's definitely good if the installation gets more robust, but maybe we can also > dig into the root cause of the random network failures. I never experienced > such constant problems with other providers, do you know anything what > could cause this? Should we consider to provide a more stable distribution > way like e.g. Github Releases? Network failures are normal, everyone has to properly deal w/ them. Our binaries are already hosted on S3 (that's where Github Releases come from as well).

Would you want to review this? https://github.com/dlang/installer/pull/187

Commits pushed to master at https://github.com/dlang/installer https://github.com/dlang/installer/commit/2f1ec46aaae444f3b0a10dde18517d7e98145ee7 fix Issue 16349 - better curl retry for install.sh script - curl's --retry option isn't really helpful b/c no timeouts are activated by default - use a retry loop with increasing sleep times instead - enable connection (5s) and download (<1KB/s for 30s) timeouts (using --max-time is too tricky b/c of the unknown download sizes) https://github.com/dlang/installer/commit/c777a91b353cc3e2daed2460a43e16420005bbd2 Merge pull request #187 from MartinNowak/fix16349 fix Issue 16349 - better curl retry for install.sh script

> Our binaries are already hosted on S3 (that's where Github Releases come from as well). Oh, but for Github Releases traffic is _free_, whereas if I am not mistaken you have to pay quite a lot to AWS. A simple estimate goes like this: 25 (size of release in MB, depends of course) * 10_000 (number of monthly downloads, I hope it's higher, see below) / 1_000 (convert to GB) * 0.09 (transfer costs to internet, per GB) and thus yields: 10K monthly downloads: 25 * 10000 / 1000 * 0.09 = 22.5$ 30K: 67.5$ 50K: 112.5$ 100K monthly downloads: 225$ According to http://erdani.com/d/downloads.daily.png, there are currently at about 1400 downloads per day (=42K /month), which means it's about 1.1K$ a year. I estimate the bill to be 30-40% less, because the Linux archives are better compressed and it's mostly those that are constantly downloaded for CI. It's still quite a lot of money (if I am not mistaken with my estimate).

Commits pushed to stable at https://github.com/dlang/installer https://github.com/dlang/installer/commit/2f1ec46aaae444f3b0a10dde18517d7e98145ee7 fix Issue 16349 - better curl retry for install.sh script https://github.com/dlang/installer/commit/c777a91b353cc3e2daed2460a43e16420005bbd2 Merge pull request #187 from MartinNowak/fix16349