Re: Intermittent pbulk build fail of editors/vim-share

["J. Lewis Muir" <jlmuir%imca-cat.org@localhost>]

On 11/15, Joerg Sonnenberger wrote:
> On Tue, Nov 15, 2016 at 11:42:16AM -0600, J. Lewis Muir wrote:
> > ==========
> > ===> Building for vim-share-7.4.1987
> > --- objects ---
> > --- auto/osdef.h ---
> > --- objects/os_macosx.o ---
> > --- objects/os_mac_conv.o ---
> > --- objects ---
> > mkdir objects
> > --- auto/osdef.h ---
> > CC="clang -Iproto -DHAVE_CONFIG_H   -I/usr/include -DMACOS_X_UNIX    " srcdir=. sh ./osdef.sh
> > --- objects/os_macosx.o ---
> > clang -c -I. -Iproto -DHAVE_CONFIG_H   -I/usr/include -DMACOS_X_UNIX  -O2 -pipe -I/usr/include -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1       -o objects/os_macosx.o os_macosx.m
> > --- objects/os_mac_conv.o ---
> > clang -c -I. -Iproto -DHAVE_CONFIG_H   -I/usr/include -DMACOS_X_UNIX  -O2 -pipe -I/usr/include -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1       -o objects/os_mac_conv.o os_mac_conv.c
> > --- objects/os_macosx.o ---
> > error: unable to open output file 'objects/os_macosx.o': 'No such file or directory'
> > 1 error generated.
> > --- objects/os_mac_conv.o ---
> > error: unable to open output file 'objects/os_mac_conv.o': 'No such file or directory'
> 
> From the output it clearly looks like it is trying to run "objects" and
> "objects/os_macosx.o" targets in parallel. If the former doesn't finish
> before the latter, the problem as seen happens. I've seen a couple of
> broken packages like this.

Hi, Joerg!

I think your analysis is spot on!  With that, I was able to reproduce
the problem as shown below.

The vim74/src/Makefile contains targets like this (among others):

==========
objects:
	mkdir objects

objects/os_macosx.o: os_macosx.m
	$(CCC) -o $@ os_macosx.m

objects/os_mac_conv.o: os_mac_conv.c
	$(CCC) -o $@ os_mac_conv.c
==========

So, none of the object file targets have a dependency on the objects
target that creates the objects directory in which the object files are
supposed to be created.

I added a delay to the objects target like this to try to reproduce the
problem:

==========
objects:
	echo XXX sleeping for 5 seconds
	sleep 5
	mkdir objects
==========

And sure enough, for a parallel make, that fails every time with the
same error I initially reported.

If I force a serial make (i.e., "bmake MAKE_JOBS_SAFE=no build"), then
it builds correctly even with the above "sleep 5" in the objects target
commands.

What's the best way to fix this?

Obviously, MAKE_JOBS_SAFE=no could be added to the package Makefile to
force a serial make; problem solved.  Would that be good?

If keeping support for a parallel make is desirable, then I'm not
sure about the best way to do it.  I would think creating a directory
before creating files inside of it is a common build case, so what's
best-practice for doing that for a parallel make?

I initially thought to add the objects target as a dependency of the .o
targets, and change the mkdir to add a -p option since the directory can
already exist:

==========
objects:
	mkdir -p objects

objects/os_macosx.o: os_macosx.m objects
	$(CCC) -o $@ os_macosx.m

objects/os_mac_conv.o: os_mac_conv.c objects
	$(CCC) -o $@ os_mac_conv.c
==========

But I'm not sure this is correct, and I don't know that the -p option to
mkdir is considered portable.

I guess the -p option could be removed with something like this:

==========
objects:
	mkdir objects 2> /dev/null || test -d objects
==========

There's also GNU Make's concept of an order-only prerequisite:

  https://www.gnu.org/software/make/manual/make.html#Prerequisite-Types

But if Vim does not require GNU Make, it seems unlikely the project
developers would be OK making it a requirement.

NetBSD's make(1) has a special .ORDER target, but it seems unlikely the
Vim project developers would be OK making NetBSD's Make a requirement,
either.

Thanks!

Lewis