Re: make: ignore stale .depend

[David Holland <dholland-tech%netbsd.org@localhost>]

On Sat, Nov 27, 2010 at 09:38:17AM -0800, Simon J. Gerraty wrote:
 >> The real problem is that the list of files on the right-hand-side of a
 >> .depend line is a derived piece of data, it's out of date with respect
 >> to its source information, and it's not getting rebuilt at the right
 >> time.
 > 
 > Agreed.

This is not, however, the whole problem; part of the problem is that
unless you're very careful saying "the list of files is out of date"
is not quite the same as saying "foo.d is out of date".

That is, suppose you have ../foo/foo.c that uses ../foo/foo.h and
../include/stupid.h, and stupid.h gets removed. Now, if foo.d does
*not* depend on stupid.h, you lose if include statements in stupid.h
change. So somewhere you must have

   foo.d: ../foo/foo.c ../foo/foo.h ../include/stupid.h

and (in the absence of either special hacks or further refinement) if
you remove stupid.h this will not rebuild foo.d but instead fail with
"make: don't know how to make ../include/stupid.h".

In order to make this work I think make not only has to be able to
rebuild and reload include files, but it has to keep track of which
dependency information came from which include files and suppress it
when those files are out of date. Then it will find that foo.d is out
of date because of stupid.h, but cause itself to forget the reference
to stupid.h so it doesn't get as far as discovering it has no rules
for regenerating stupid.h. This will be fairly delicate to implement
correctly, especially with the mess make currently is inside. :(

 > As I've noted before, generating that data as a side effect of
 > compilation mitgates a lot of the issues.

No, on the contrary I'd say it doesn't, and in fact this problem only
appears when you try to do depends automatically. If you have manual
depends, removing stupid.h will cause "don't know how to make
stupid.h" but then you just make depend and all is henceforth well.

It's when you try to do it automatically that make ends up unable to
proceed because it can't regenerate stupid.h.

 > This problem is also pretty much solved in meta mode.

I'm not convinced that meta mode is (or can be) any different than gcc
-MD for these purposes; all it can do is record which files were read
during compilation, which leads to the same problem if one of them
disappears later.

 > >The only proper fix is to express that dependency correctly and let
 > >the normal mechanisms of make take care of it. 
 > 
 > To some extent, yes.  Make doesn't however have a mechanism for
 > forgetting nodes and recomputing them, so you still face the issue of it
 > learning a bogus dependency and being stuck with it.

Well yes, the internals of make need to be strengthened. I've been
saying that for years now and maybe I'll eventually get to it.

We also need a way to write a rule like

   foo.o foo.d: ../foo/foo.c ../foo/foo.h ../include/stupid.h
           $(CC) $(CFLAGS) -c -MD ../foo/foo.c

that tells make that the two targets on the left are produced jointly
by running the recipe once. (If you just write it this way it will
currently work in some cases and not others.)

 > The current FROM_DEPEND etc flag at least allows make to discount such
 > dependencies. 

Yes, but can we please make it an explicit attribute instead of a
special-case hack for .depend? That is, something like

   .MISSING: ../foo/foo.c ../foo/foo.h ../include/stupid.h

which mkdep can easily put into the .d and/or .depend files.

 > >gmake has a feature that lets it rebuild and rescan included makefile
 > >fragments that are out of date; ISTM that this, combined with correct
 > >rules for the .depend and *.d files (which we do not currently have),
 > >should solve the problem completely and permanently.
 > 
 > The suggested way to use that is much the same as my autodep.mk
 > and the auto-regeneration can lead to problems too.

Yes, well, gmake lacks other necessary features for doing things
right, and the suggestions in the gmake manual are frequently wrong
anyway.

 >> There is no need to treat rules read from .depend in a special way and
 >> we really ought to remove that logic.
 > 
 > Assuming you have a better solution implemented, sure (and I'm not
 > convinced gmake's example counts).

Of course it doesn't count, but what about making it an explicit
attribute?

 > Until then though the current hacks are better than nothing.

I'm not certain of that -- they'll end up needing to be supported in
the long term. There's enough of that in make already...

-- 
David A. Holland
dholland%netbsd.org@localhost