Bug 13946 – ddox should rebuild documentation incrementally

A typical flow in building dlang.org is (a) modify one file in phobos or druntime, (b) rebuild the site and upload. Right now we don't have a mechanism to rebuild only the appropriate ddox outputs, so we need to rebuild from a clean slate, which takes a long time. I'm unclear on what would be a good approach because I'm not fluent with dub. One idea would be to have it process one module in one command; then dependencies are easy to track and make -j can easily parallelize that.

There are currently three parts that take considerable time: 1. compiling the "dpl-docs" executable 2. compiling Phobos+Druntime with "-o- -X -D" to generate the JSON file 3. generating the actual HTML files The first step is currently slow because DUB is executed with --force (thus rebuilding it on every run) to work around <https://github.com/D-Programming-Language/dub/issues/331>. This is fixed for the current RC of DUB (will be tagged final tomorrow), so that we can remove the --force now. The second step is relatively fast, so we can probably keep it as it is for now. For the third step I've implemented conditional writing of HTML files using a cache file that keeps track of the MD5 sum of all written files. As a side effect it now also deletes obsolete HTML files of earlier runs. This is in DDOX 0.10.6.

I guess this can be closed now. 1: is solved (the --force workaround has been removed) 2: remains, but is fast. 3: is solved