Re: bsd.lib.mk question

[Simon Gerraty <sjg%juniper.net@localhost>]

The following is an extract from the paper I presented at BSDCan,
seems relevant to this thead:

>Going fast though, doesn't matter if you get incorrect results.
>
>Bmake works in two modes, *jobs* mode and *compat* mode.
>In compat mode, targets are evaluated depth first in the order listed.
>In jobs mode, targets are evaluated breadth first and in parallel.
>
>For example::
>
>       LIB = fool
>
>       SRCS = parser.c file1.c file2.c file3.c
>
>       parser.c: parser.y
>               ${YACC} -d -o ${.TARGET} ${.IMPSRC}
>               mv t.tab.y ${.TARGET:T:R}.h
>
>       .include <bsd.lib.mk>
>
>In compat mode, everything *just works*, because the SRCS are
>built sequentially - in the order listed.  So everything associated
>with producing ``parser.o`` will happen before ``file1.o`` is attempted.
>
>In jobs mode (e.g. ``mk -j8``), things can rapidly fall apart.
>With just the information listed, each of the SRCS will be compiled in
>parallel.  If any of ``file*.c`` includes ``parser.h``, then success
>will be random depending on whether the ``mv`` step of the script
>above completes before any of the compilations of ``file*.c`` need it.
>This is called a race condition and  more available CPUs typically
>exacerbates the situation.
>
>To avoid that race condition, more explicit dependencies are needed.
>This however, is often done incorrectly::
>
>       # wrong: this can cause YACC to be run twice - at the same time!
>       parser.c parser.h: parser.y
>               ${YACC} -d -o ${.TARGET:T:R}.c ${.IMPSRC}
>               mv t.tab.y ${.TARGET:T:R}.h
>
>       file1.o:        parser.h
>
>The above code introduces a different race condition - both
>``parser.c`` and ``parser.h`` can trigger running the same script
>(at about the same time) which usually ends in tears.
>
>The next attempt might be::
>
>       # wrong: likelihood of circular dependencies
>       parser.h: parser.c
>       parser.c: parser.y
>               ${YACC} -d -o ${.TARGET:T:R}.c ${.IMPSRC}
>               mv t.tab.y ${.TARGET:T:R}.h
>
>       file1.o:        parser.h
>
>but that can result in cyclic dependencies if ``parser.c`` includes
>``parser.h`` and this ends up reflected in ``.depend``.
>
>The best bet would be::
>
>       parser.h: parser.y
>               ${YACC} -d -o ${.TARGET:T:R}.c ${.IMPSRC}
>               mv t.tab.y ${.TARGET}
>
>       parser.c:       parser.h
>
>       file1.o:        parser.h
>
>This avoids the race conditions as well as the chance of cyclic
>dependencies.
>
>In the current Junos build, leaf makefiles do not run in *jobs* mode
>by default.  So the parallelism is mainly at the top-level.
>For small leaf directories this does not matter.
>
>For big libs like ``libc`` it does matter.
>There is therefore a ``USE_JOBS`` knob that can be set in a makefile,
>to indicate that it can build in *jobs* mode, and this gets picked up
>by the top-level (via the clues printed by ``mk dpadd``), so all is
>not lost.  The speedup for big libs like ``libc`` is significant.
>
>In *meta* mode, we capture in the ``Makefile.depend*``
>sufficient information to ensure a successful parallel build in a
>clean tree.  Thus the default will be to allow leaf makefiles to build
>in *meta jobs* mode.
>
>Even better, you can do a simple ``mk`` in *compat* mode while
>bootstrapping a new makefile, to capture the local dependencies, and
>it will then generally *just work* in *jobs* mode.