Bug 13517 – Only touch .di generated file if changed

Running dmd -H always generates a fresh .di file, even if an existing .di file exists and has identical content. This confuses build tools - such as make and derivatives - that rely on file timestamp to assess what needs to be rebuilt. The compiler should generate the header in e.g. xyz.di.tmp file. Then, if xyz.di exists, compare the content of xyz.di with xyz.tmp. If they compare equal, remove xyz.di.tmp and otherwise rename it to xyz.tmp.

but why compiler should do that? it's a build tool task. the build tool should know that .di file is generated from .d and invoke generator only if .di file is missing or older than .d.

Hoisting the approach in the compiler will make it easier for people using build tools.

(In reply to Ketmar Dark from comment #1) > but why compiler should do that? it's a build tool task. the build tool > should know that .di file is generated from .d and invoke generator only if > .di file is missing or older than .d. That idea only applies if you want to do .di generation as a separate compiler invocation from compilation.

(In reply to Vladimir Panteleev from comment #3) > That idea only applies if you want to do .di generation as a separate > compiler invocation from compilation. and why you wouldn't, having the build tool? speed gain is neglible, and compiler gains new useless code.

Again: hoisting this into the compiler lowers the barrier of entry and simplifies the life of anyone building D code with .di files.

The generated object file isn't compared against the existing one, why should other generated files? It seems strange to me to make the .di generation special in this way.

(In reply to Andrei Alexandrescu from comment #5) > Again: hoisting this into the compiler lowers the barrier of entry and > simplifies the life of anyone building D code with .di files. again: no, it's not.

(In reply to Walter Bright from comment #6) > The generated object file isn't compared against the existing one, why > should other generated files? It seems strange to me to make the .di > generation special in this way. Thanks for asking. The difference here is that only the linked binary depends on the .o file, whereas a lot more build artifacts depend on the .di file.

(In reply to Ketmar Dark from comment #7) > (In reply to Andrei Alexandrescu from comment #5) > > Again: hoisting this into the compiler lowers the barrier of entry and > > simplifies the life of anyone building D code with .di files. > again: no, it's not. Just stating it won't make it true. I'd appreciate if you stopped these content-less comments, they are disruptive and paralyze any progress in the matter. Feel free to substantiate the points you're making. Thanks.

(In reply to Andrei Alexandrescu from comment #9) > Just stating it won't make it true. same for you. all this conversation boils down to "i believe that this will help." "and i don't." "and i believe." "and i don't." what build tools do you mean? is there any sane build tool that can't cope with (re)generating ".di" from ".d"? why compiler should do the work for which build tools were built? it's compiler work to (re)generate everything we asked it to. and it's build tool's work to minimize compiler invocations by doing various checks. good build tool is perfectly able to notice that we need both new ".di" and new ".o" files and do it with one exec(). also imagine that i'm doing "force rebuilding". we'll need new command line flag that which tells dmd to regenerate .di file even if it's source .d file was not changed. sure we can workaround that by teaching build tool to call 'touch' or something, but for what reason? either we have full-blown build tool in the compiler itself or we don't try to teach compiler some tricks which build tool should do and then start fighting with compiler to stop it tricking. i'm on for the concept of "cached ASTs" (something like delphi's .dcu files) and teaching dmd to rebuild all changed modules and updating cached ASTs. this will greatly imporoves compilation speed (no need to do lexing, parsing and semantic analyzis again and again) and will become a nice and simple build system. but this is a very complex change and it's completely independent of ones you proposing.

(In reply to Ketmar Dark from comment #10) > (In reply to Andrei Alexandrescu from comment #9) > > Just stating it won't make it true. > same for you. all this conversation boils down to "i believe that this will > help." "and i don't." "and i believe." "and i don't." I've explained my reasons: it's the right place to put this kind of logic because it simplifies the use of .di files for everyone. > what build tools do you mean? is there any sane build tool that can't cope > with (re)generating ".di" from ".d"? why compiler should do the work for > which build tools were built? There is indeed some complication. make must be invoked two times in order to first generate and then use the dependencies. This is because make has separate stages for reading dependencies and building stuff; at work we used to drive two calls to make, but more recently we have a build system (actually two) that know how to maintain dependencies dynamically. It is customary for today's advanced languages to provide dependency information for build tools, and D is no exception. Maintaining dependencies manually by tools that are poorly suited to grok language semantics does not scale well. > it's compiler work to (re)generate everything we asked it to. and it's build > tool's work to minimize compiler invocations by doing various checks. good > build tool is perfectly able to notice that we need both new ".di" and new > ".o" files and do it with one exec(). This view is rather simplistic. The "various checks" depend on how the language manages dependencies and need to be helped by language-specific tools, of which the most obvious is the compiler itself. > also imagine that i'm doing "force rebuilding". we'll need new command line > flag that which tells dmd to regenerate .di file even if it's source .d file > was not changed. sure we can workaround that by teaching build tool to call > 'touch' or something, but for what reason? A force rebuild would start by deleting generated .di files. > either we have full-blown build tool in the compiler itself or we don't try > to teach compiler some tricks which build tool should do and then start > fighting with compiler to stop it tricking. This is simple correct generated file management: what doesn't change doesn't get touched. I don't understand the agitation around this. > i'm on for the concept of "cached ASTs" (something like delphi's .dcu files) > and teaching dmd to rebuild all changed modules and updating cached ASTs. > this will greatly imporoves compilation speed (no need to do lexing, parsing > and semantic analyzis again and again) and will become a nice and simple > build system. but this is a very complex change and it's completely > independent of ones you proposing. I encourage you to submit that as a separate enhancement request for that. But please stop bullying this into the ground. I don't have the time to defend it and I have no interest in "winning" the argument consider you've won; I do have skin in the game and need to see this through. On the other hand it's relatively easy for you to naysay this by brute force. Clearly we could talk at great lengths about the relative responsibilities of compilers and external build tools, seeing as it's a matter in which reasonable people do disagree. Also scripting around the language is a definite possibility. But a stalemate here leads to the net result that nothing gets done and there's no progress. So please don't make this all into a net negative. Thanks.

(In reply to Andrei Alexandrescu from comment #11) > I've explained my reasons no, you don't. you just keep saying "it will simplify everyone's life". and when i'm saying that it will not (using your own style of argumentation, actually), you accused me of bullying. i don't want to talk on this issue anymore, it's meaningless. i don't see any way of making discussion productive using such "no-op arguments".

(In reply to Ketmar Dark from comment #12) > (In reply to Andrei Alexandrescu from comment #11) > > I've explained my reasons > no, you don't. you just keep saying "it will simplify everyone's life". My original argument: "Running dmd -H always generates a fresh .di file, even if an existing .di file exists and has identical content. This confuses build tools - such as make and derivatives - that rely on file timestamp to assess what needs to be rebuilt." Your conter-argument has been that that's the responsibility of the build tool. That's true, but it makes creating build tools for D more elaborate and more complicated than it should. My take is D should offer a simpler and better interface to classic build tools that consult timestamps and rely on the compiler to provide dependency information. I'll elaborate below. > and > when i'm saying that it will not (using your own style of argumentation, > actually), you accused me of bullying. With respect, that's exactly what happens. This is disproportionate response to a simple matter. > i don't want to talk on this issue anymore, it's meaningless. i don't see > any way of making discussion productive using such "no-op arguments". Let me elaborate how I see this would help building D projects. Currently (incremental builds, simple dependency management, no .di files), following the touch of xyz.d: 1. The deliverable depends on xyz.o and xyz.o depends on xyz.d. Therefore xyz.o is rebuilt. 2. The dependencies (previously saved with dmd -deps) list xyz.d as a dependence for all other modules that transitively import it. The timestamp of xyz.d has changed. Therefore the transitive closure of modules importing xyz must be rebuilt. 3. Finally deliverable(s) are linked etc. With .di files and without any special steps, the above doesn't change /because/ the .di files will be touched during the building of xyz.o. (There is a change in the sense that now modules depend on .di files, not .d files.) Now consider there's scripting around the build system that only touches xyz.di if it has new content. In that case, only a subset of changes to xyz.d will be reflected in xyz.di. So the process would go: 1. The deliverable depends on xyz.o and xyz.o depends on xyz.d. Therefore xyz.o is rebuilt. 2. The dependencies (previously saved with dmd -deps) list xyz.di as a dependence for all other modules that transitively import it. The timestamp of xyz.di may or may not have changed. In particular, changes to implementation of non-generic functions don't affect the interface. In that case there's no ripples. 3. Finally deliverable(s) are linked etc. This is going to go the same for all timestamp-based build systems so it's somewhat generic. The bit about touching the .di files conditionally is more D-specific. Now the question is where to place that code. Clearly the build tool could muster it. In my opinion it's easier to interface with generic build tools if the timestamps just work, as opposed to describing specifics on how these could be made to work. I'll also note that the story with .o files is quite different but may be worth looking at in the context of D. There are just few changes that don't affect .o file generation. The two that I can think of are comments and bodies of templates that aren't used within the same file. If we conclude these are somewhat frequent, conditional touching of the .o file may be interesting to do.

(In reply to Andrei Alexandrescu from comment #13) > With respect, that's exactly what happens. This is disproportionate response > to a simple matter. this proposal leads to increasing complexity of compiler code. and it's for nothing, actually, 'cause this work can be easily done with external script or with build tool itself if it's not 'make'. i'm just strogly against: 1. adding unnecessary code to compiler. 2. turning dmd to half-baked build tool. > Now consider there's scripting around the build system that only touches > xyz.di if it has new content. then we have strange hidden dependency: .di file *seems* to be dependend of .d file, yet it is clearly older than .d file and somehow it's not regenerated (why?). so the build tool which relies on timestamps is either confused or the user need to remember the dependency but don't inform build tool about it. i see only added complexity here. such complexity is unnecessary for small projects with small build times, and it can easily turn to complete disaster with big complex projects (been there, seen that). complex projects *will* fight with this feature if their build tools are timestamp-based (or build process will *inevitably* go out of control). and it's really easy to do this with simple comparison script (ah, just do 'sort' and then calculate md5, for example). that's why i'm saying that overal win is close to nothing, but compiler implementation gains more code. all in all you aren't convinced me. but your position is clear to me now, so we can stop arguing and wait for PR. and then let Walter to decide. sorry for being rude, it was luck of clear understanding from my side.

(In reply to Vladimir Panteleev from comment #3) > (In reply to Ketmar Dark from comment #1) > > but why compiler should do that? it's a build tool task. the build tool > > should know that .di file is generated from .d and invoke generator only if > > .di file is missing or older than .d. > > That idea only applies if you want to do .di generation as a separate > compiler invocation from compilation. I was about to argue against Andrei's position, but this makes sense -- the .di has no dependencies in the build tool, nor does it know how to build the .di file. It's just a side effect of building the .o The only caveat is that you can't have the same project that generates the .di files (as a side effect) depend on the .di files in any way. Given the purpose of .di files, that seems reasonable.

(In reply to Steven Schveighoffer from comment #15) > .di has no dependencies in the build tool, nor does it know how to build the > .di file. It's just a side effect of building the .o ahem. just tried `dmd -c z00.d`. no .di file was built. what kind of "side effect" are you talking here? ah, and another try: `dmd -c -o- -H z00.d`. this time i got .di file, but no .o file. i'm confused. and why build tool doesn't know about '.di'? any sane build tool allows adding dependencies. and no build tools except DUB and my own knows about D out-of-the-box anyway.

(In reply to Ketmar Dark from comment #16) > ahem. just tried `dmd -c z00.d`. no .di file was built. what kind of "side > effect" are you talking here? ah, and another try: `dmd -c -o- -H z00.d`. > this time i got .di file, but no .o file. i'm confused. -c builds .o -o- suppresses .o file. -H builds.di file. So: Stevens-MacBook-Pro-2:testd steves$ ls testdi.* testdi.d Stevens-MacBook-Pro-2:testd steves$ cat testdi.d void foo() {} Stevens-MacBook-Pro-2:testd steves$ dmd -c -H testdi.d Stevens-MacBook-Pro-2:testd steves$ ls testdi.* testdi.d testdi.di testdi.o > > and why build tool doesn't know about '.di'? any sane build tool allows > adding dependencies. and no build tools except DUB and my own knows about D > out-of-the-box anyway. I mean, the build tool knows it only as a dependency, not as a target. It has no idea how to build the .di file. The .di file is built as a side effect of building the .o file. But as I said, you would need to ensure the project that builds the .di file cannot depend on the .di file, because it doesn't know how to build them, it just does so as a side effect.

I would add a stipulation that if the purpose of the compiler invocation is ONLY to build the .di file, then the compiler should rewrite it. So dmd -c -o- -H should still touch the .di file even if it's identical to the existing one. Otherwise, this disfavors a build system where the .di is built as a separate step (and causes a continual "rebuild" of a .di file every time the build is called).

(In reply to Steven Schveighoffer from comment #17) > -c builds .o Of course, this is wrong :) -c suppresses link.

(In reply to Steven Schveighoffer from comment #17) so it's not a "side effect", .di file has to be explicitly asked. > > and why build tool doesn't know about '.di'? > I mean, the build tool knows it only as a dependency, not as a target. It > has no idea how to build the .di file. The .di file is built as a side > effect of building the .o file. no, it's not a "side effect". and build tool doesn't know how to build .o from .d too. one must provide the rule which builds .o from .d. one must provide the rule which builds .di from .d. simple and easy. > But as I said, you would need to ensure the project that builds the .di file > cannot depend on the .di file, because it doesn't know how to build them, it > just does so as a side effect. if project uses generated files that eventually needs regeneration and it's build system doesn't know about that files, this project's build scripts are foobared. that is exactly the mess that will lead to complete disaster sooner or later.

(In reply to Steven Schveighoffer from comment #18) > I would add a stipulation that if the purpose of the compiler invocation is > ONLY to build the .di file, then the compiler should rewrite it. So dmd -c > -o- -H should still touch the .di file even if it's identical to the > existing one. > > Otherwise, this disfavors a build system where the .di is built as a > separate step (and causes a continual "rebuild" of a .di file every time the > build is called). Wouldn't that be easily achieved by "rm -rf generated/*.di" at the beginning of that separate step?

(In reply to Ketmar Dark from comment #20) > (In reply to Steven Schveighoffer from comment #17) > so it's not a "side effect", .di file has to be explicitly asked. > > > > and why build tool doesn't know about '.di'? > > I mean, the build tool knows it only as a dependency, not as a target. It > > has no idea how to build the .di file. The .di file is built as a side > > effect of building the .o file. > no, it's not a "side effect". and build tool doesn't know how to build .o > from .d too. one must provide the rule which builds .o from .d. one must > provide the rule which builds .di from .d. simple and easy. Actually it's not that simple and easy. One invocation of the compiler produces both .o and .di files. A common mistake made with rules that generate multiple files is to do https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html, which is wrong. There are a few better solutions, neither of which trivial. See e g. http://www.cmcrossroads.com/article/rules-multiple-outputs-gnu-make.

(In reply to Andrei Alexandrescu from comment #21) > > Wouldn't that be easily achieved by "rm -rf generated/*.di" at the beginning > of that separate step? Sure, but "easily achieved" is not "easily explained" or "easily justified" It is very counterintuitive to have a tool that is resolving dependencies to require an extra step to update the date of file. Especially one that does not act that way for any other generated files. I will note, this will break a LOT of existing projects. Including druntime. (In reply to Ketmar Dark from comment #20) > (In reply to Steven Schveighoffer from comment #17) > so it's not a "side effect", .di file has to be explicitly asked. Perhaps you are getting caught up in terminology. It's not a target the build system is trying to generate. Whatever term you want to use for that, use that. > no, it's not a "side effect". and build tool doesn't know how to build .o > from .d too. one must provide the rule which builds .o from .d. one must > provide the rule which builds .di from .d. simple and easy. It knows how to build the .o because we tell it. E.g.: %.o : %.d dmd -c -H %.d (please note, not a makefile expert, so the above may not be valid make syntax, but I hope you get the idea) But my point is, you don't tell it about building the .di file because the build of the .di file is implicit in the rule for the .o. > > > But as I said, you would need to ensure the project that builds the .di file > > cannot depend on the .di file, because it doesn't know how to build them, it > > just does so as a side effect. > if project uses generated files that eventually needs regeneration and it's > build system doesn't know about that files, this project's build scripts are > foobared. that is exactly the mess that will lead to complete disaster > sooner or later. But your project doesn't know how to build /usr/include/stdio.h, so why should your project care about how some other project's .di file is made? It requires some manual ordering. For instance, you can't just treat all project files as individuals, you have to build all files for project a first if project b depends on a. One thing that would make this scheme fail is if one removes the .di file and expects it to regenerate. But I think this is a caveat of an uncommon build system, you just have to live with that. I will say, I think the enhancement is unusual and counterintuitive. It may be best to enable this mechanism with a special flag. Maybe like -H+

(In reply to Steven Schveighoffer from comment #23) > Perhaps you are getting caught up in terminology. It's not a target the > build system is trying to generate. so it should *never* be (re)generated. > But my point is, you don't tell it about building the .di file because the > build of the .di file is implicit in the rule for the .o. and build system doesn't even know that we are using implicitly regeneration for some files and that other files can depend of this unknown files. it's a very… strange way to using build system. > But your project doesn't know how to build /usr/include/stdio.h, so why > should your project care about how some other project's .di file is made? i never need to (re)generate /usr/include/stdio.h, it's god-given. and i'm assuming that we are talking about .di files that can be generated by various parts of the project we are trying to build. god-given .di files can be treated just like <stdio.h> — i.e. not mentioned at all. and not silentely regenerated under any circumstances. > I will say, I think the enhancement is unusual and counterintuitive. It may > be best to enable this mechanism with a special flag. Maybe like -H+ hm. if new behavior will not be default one and must be explicitly turned on, i'm all in for this ER, it will not hurt.

(In reply to Ketmar Dark from comment #24) > (In reply to Steven Schveighoffer from comment #23) > > Perhaps you are getting caught up in terminology. It's not a target the > > build system is trying to generate. > so it should *never* be (re)generated. No, it will only be regenerated if different. But the build system isn't involved in deciding when and where to build it. It just is concerned with the .o, and the compiler builds the .di according to the build line (and these funky rules). > > But my point is, you don't tell it about building the .di file because the > > build of the .di file is implicit in the rule for the .o. > and build system doesn't even know that we are using implicitly regeneration > for some files and that other files can depend of this unknown files. it's a > very… strange way to using build system. It's strange, but not the strangest I have seen. I won't say I think it's the *best* way to do this, but the logic seems sound to me. > > > But your project doesn't know how to build /usr/include/stdio.h, so why > > should your project care about how some other project's .di file is made? > i never need to (re)generate /usr/include/stdio.h, it's god-given. An importing project does not know how to generate foreign .di files that it depends on, so they are "god-given" also. A project typically shouldn't depend on .di files inside itself, it has the .d files, and you lose a lot of compiler nifty things, such as inlining and CTFE, when you use .di files instead of .d

I agree with Walter. Why should the compiler care if the previous version of the .di file different or not? That's not the compiler's job. If the build tool can't properly handle files that don't change (and I know make doesn't 'cos make sux), then it should be replaced with a saner build tool.

(In reply to hsteoh from comment #26) > I agree with Walter. Why should the compiler care if the previous version of > the .di file different or not? That's not the compiler's job. If the build > tool can't properly handle files that don't change (and I know make doesn't > 'cos make sux), then it should be replaced with a saner build tool. I understand. One way to look at this (which is what motivated the approach) is from the perspective of automating maintenance of C++ xyz.h/xyz.cpp files - which is a noble goal and the stated purpose of .di generation. When a C++ programmer needs to change the inline/template/interface of a module, they'll change xyz.h. Then the build system will notice the updated timestamp and proceed with building dependent files. For changes to non-inline non-template functions etc, the programmer only touches xyz.cpp and no other dependents are rebuilt. Switch over to D-land, where the story with xyz.di/xyz.d is similar EXCEPT now the former is automatically taken care of; all the programmer needs to do is update xyz.d and then the .di file will be changed (or not) accordingly. Touching the .di file only when necessary is the natural automation of a process in which changing the interface vs. the implementation of a module affect different parts of the deliverables.

Relying solely on timestamp to detect change is an antiquated concept that basically only make (and misguided attempts to build on it) suffers from. Modern build systems do not have this problem. But given that it's not an ideal world (otherwise we wouldn't need to interface with C++ to begin with :-P) one possible workaround is to write your compile rules something along these lines: sucky.o sucky.di: sucky.d $(DMD) $(DFLAGS) sucky.d -ofsucky.o -H -Hfsucky_new.di cmp sucky.di sucky_new.di ; if [ $? -ne 0 ] ; then cp sucky_new.di sucky.di ; fi

(In reply to hsteoh from comment #28) > Relying solely on timestamp to detect change is an antiquated concept that > basically only make (and misguided attempts to build on it) suffers from. > Modern build systems do not have this problem. I agree many of the newer build systems rely on hashing the contents of files. But not all. > But given that it's not an ideal world (otherwise we wouldn't need to > interface with C++ to begin with :-P) one possible workaround is to write > your compile rules something along these lines: > > sucky.o sucky.di: sucky.d > $(DMD) $(DFLAGS) sucky.d -ofsucky.o -H -Hfsucky_new.di > cmp sucky.di sucky_new.di ; if [ $? -ne 0 ] ; then cp sucky_new.di > sucky.di ; fi This is the best illustration on why we shouldn't inflict that on the user. You have two bugs and one extra issue in there.

THIS ISSUE HAS BEEN MOVED TO GITHUB https://github.com/dlang/dmd/issues/17683 DO NOT COMMENT HERE ANYMORE, NOBODY WILL SEE IT, THIS ISSUE HAS BEEN MOVED TO GITHUB