Bug 16746 – Please output Makefile-style depfiles for ninja and make

GCC and other compilers emit dependency information as Make-style depfiles, allowing Make and make-replacements like Ninja to recompile dependencies as soon as some unit higher up in the dependency chain has been changed. This is tremendously useful, especially in D, where descending units have to be rebuild e.g. on a template change or function signature change. Unfortunately, DMDs `-deps=` flag makes it output a custom format, which makes adding DMD and any compiler that is based on the DMD frontend to build systems using Ninja and Make harder than it needs to be. The Makefile-syntax used by GCC and Clang is the de-facto standard and changing all other tools to support DMD will be impossible. See https://ninja-build.org/manual.html#_depfile for a bit more information on depfiles. Unfortunately, no format documentation exists, but the format is simply a Makefile without variables in the form of: ``` Make the_generated_file.o: source1.d source2.d source3.d ``` Ninja's tests also contain a bit of information: https://github.com/ninja-build/ninja/blob/master/src/depfile_parser_test.cc Outputting both formats would be helpful, GDC has `-fdeps=` and `-fmake-deps=` for that (with the latter option leading to a crash unfortunately). Cheers, Matthias See also the original bug report against LDC: https://github.com/ldc-developers/ldc/issues/1802

(In reply to Matthias Klumpp from comment #0) > Outputting both formats would be helpful, GDC has `-fdeps=` and > `-fmake-deps=` for that (with the latter option leading to a crash > unfortunately). Where's the bug report?

(In reply to Iain Buclaw from comment #1) > (In reply to Matthias Klumpp from comment #0) > > Outputting both formats would be helpful, GDC has `-fdeps=` and > > `-fmake-deps=` for that (with the latter option leading to a crash > > unfortunately). > > Where's the bug report? I didn't create one yet because I was under time pressure when finding that out - and for a proper bug report I would need a bit more information (or else the first question after telling "it crashes" would be "can you provide an example", since a simple "Hello world" doesn't seem to trigger this).

@Matthias a GDC test case would be really appreciated. This should be easily reducible with Dustmite https://github.com/CyberShadow/DustMite . But if you don't have the time to reduce the code you can also send the complete code to my email address. Alternatively if it's a OSS project anyway just provide some instructions to reproduce the problem and I'll get a reduced test case.

Matthias, have you seen rdmd's --makedepend flag? I think it may do what you need.

(In reply to Johannes Pfau from comment #3) > @Matthias a GDC test case would be really appreciated. This should be easily > reducible with Dustmite https://github.com/CyberShadow/DustMite . But if you > don't have the time to reduce the code you can also send the complete code > to my email address. Alternatively if it's a OSS project anyway just provide > some instructions to reproduce the problem and I'll get a reduced test case. Looks like this is all resolved with GDC 6 \o/ - I can't reproduce any of the issues and -fmake-deps seems to work pretty well, unless https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=807491 is still a thing. Anyway, I submitted a patch to Meson to use this by default for GDC now: https://github.com/ximion/meson/commit/0deb7a9e0c5e358dc7d1d07d8ac10b5c82ce33cd

(In reply to Vladimir Panteleev from comment #4) > Matthias, have you seen rdmd's --makedepend flag? I think it may do what you > need. This is indeed what I am looking for - does dmd have this too and I am just not seeing it? (dmd --help didn't show something useful)

> This is indeed what I am looking for - does dmd have this too and I am just not seeing it? (dmd --help didn't show something useful) No - it's implemented here: https://github.com/dlang/tools/blob/master/rdmd.d#L346 You package `rdmd` as well, right? Or is there a problem with using `rdmd`?

(In reply to greenify from comment #7) > [...] > You package `rdmd` as well, right? No, there is nothing using it in Debian (and we already lack manpower for the existing packages). > Or is there a problem with using `rdmd`? Yes, it doesn't support all command-line flags one would use with ldc, gdc or dmd. Any wrapper is not an option.

(In reply to Matthias Klumpp from comment #8) > Yes, it doesn't support all command-line flags one would use with ldc, gdc > or dmd. rdmd simply forwards flags it doesn't recognize to the compiler. > Any wrapper is not an option. Why?

(In reply to Vladimir Panteleev from comment #9) > (In reply to Matthias Klumpp from comment #8) > > Yes, it doesn't support all command-line flags one would use with ldc, gdc > > or dmd. > > rdmd simply forwards flags it doesn't recognize to the compiler. > > > Any wrapper is not an option. > > Why? Because the wrapper is not available everywhere, and going through a wrapper in order to fix a missing thing in dmd is just a workaround. DMD(FE) should be able to do this natively. When adding support to Meson/CMake/Automake/Makefiles, telling users that they need the compiler but also this other tool which - in the Meson world - has the sole purpose of supplying a missing functionality in DMD is pretty cumbersome. It's a workaround, but no real solution to the issue. At the moment, Debian/Uhuntu/Fedora/OpenSUSE and probably a couple of other don't ehip rdmd at all, which makes demanding it just to work around this bug not worthwile anyway.

(In reply to Matthias Klumpp from comment #10) > Because the wrapper is not available everywhere, and going through a wrapper > in order to fix a missing thing in dmd is just a workaround. DMD(FE) should > be able to do this natively. > When adding support to Meson/CMake/Automake/Makefiles, telling users that > they need the compiler but also this other tool which - in the Meson world - > has the sole purpose of supplying a missing functionality in DMD is pretty > cumbersome. It's a workaround, but no real solution to the issue. I'm not sure about referring to a requested feature as "missing functionality", but thanks for the clarification. Regardless, we will hopefully one day have rdmd as part of dmd itself, or use the compiler as a library, so duplicating functionality in both dmd and rdmd right now does not look very appealing; until then, you may want to look into more pragmatic solutions, such as using rdmd. > At the moment, Debian/Uhuntu/Fedora/OpenSUSE and probably a couple of other > don't ehip rdmd at all, which makes demanding it just to work around this > bug not worthwile anyway. I think rdmd should be shipped in the same package as dmd. This is how the binary packages from dlang.org for all platforms are. Picking and choosing which utilities the distribution considers useful will inevitably lead to confusion and a poorer user experience.

(In reply to Vladimir Panteleev from comment #11) > (In reply to Matthias Klumpp from comment #10) > > Because the wrapper is not available everywhere, and going through a wrapper > > in order to fix a missing thing in dmd is just a workaround. DMD(FE) should > > be able to do this natively. > > When adding support to Meson/CMake/Automake/Makefiles, telling users that > > they need the compiler but also this other tool which - in the Meson world - > > has the sole purpose of supplying a missing functionality in DMD is pretty > > cumbersome. It's a workaround, but no real solution to the issue. > > I'm not sure about referring to a requested feature as "missing > functionality", but thanks for the clarification. > > Regardless, we will hopefully one day have rdmd as part of dmd itself, or > use the compiler as a library, so duplicating functionality in both dmd and > rdmd right now does not look very appealing; until then, you may want to > look into more pragmatic solutions, such as using rdmd. Sounds really good, but AFAIK this is still in the far future... GDC supports writing depfiles properly, and having this feature in DMD would enable it for DMD and LDC at the same time, making depfiles work for all D compilers. > > At the moment, Debian/Uhuntu/Fedora/OpenSUSE and probably a couple of other > > don't ehip rdmd at all, which makes demanding it just to work around this > > bug not worthwile anyway. > > I think rdmd should be shipped in the same package as dmd. This is how the > binary packages from dlang.org for all platforms are. Picking and choosing > which utilities the distribution considers useful will inevitably lead to > confusion and a poorer user experience. Usually in distributions we package just the minimal amount of things necessary to compile a certain leaf package (e.g. all the stuff one needs to compile Vibe.d/Tilix/...) and more is added when another package requires the feature. At the moment, nothing uses rdmd, and adding it would mean someone would need to maintain the package over the lifetime of the distro release. Personally I try to avoid taking on new packages on a "could be useful" basis, because the time I can invest in maintenance is limited, and it's better to have fewer well-supported packages than lots of them which are poorly maintained. (Team maintenance helps a lot with this, but only if the team is actually big enough, which isn't the case in with D yet).

(In reply to Matthias Klumpp from comment #12) > Sounds really good, but AFAIK this is still in the far future... > GDC supports writing depfiles properly, and having this feature in DMD would > enable it for DMD and LDC at the same time, making depfiles work for all D > compilers. That's true. > Usually in distributions we package just the minimal amount of things > necessary to compile a certain leaf package (e.g. all the stuff one needs to > compile Vibe.d/Tilix/...) and more is added when another package requires > the feature. That makes sense if you treat D purely as a dependency, but I think it's worth looking at it as a software application package as well. rdmd is mentioned in TDPL, so it is essentially part of D. This is why I'm uneasy about the idea of splitting up the tools that dlang.org ships together. > At the moment, nothing uses rdmd, and adding it would mean someone would > need to maintain the package over the lifetime of the distro release. Why a separate package? Why not ship it with dmd? Is it just because it's useful not just for dmd, but also ldc/gdc? If so, why not have a dtools package like Arch Linux, which includes rdmd, dustmite, ddemangle, etc.?

I went ahead and implemented support for -M, -MD, -MF=..., -MT=... flags which generate Makefile-style depfiles from DMDFE itself. https://github.com/dlang/dmd/pull/6961

*** This issue has been marked as a duplicate of issue 14188 ***