Re: Soft link issues on APFS (Apple File System)

[Greg Troxel <gdt%lexort.com@localhost>]

Shay Perlstein <Shay.Perlstein%tomtom.com@localhost> writes:

> Note that we also see these issues when compiling Safe or with single thread.

"Safe?"  Your language is not matching pkgsrc.   Please tell us what
package you are talking about, how you have set up pkgsrc, and precisely
what commands you are running.  (If you don't want to share any of that,
you need a consultant, not pkgsrc-users.)

> At first building graphviz was a challenge, it failed very often, on several machines. So we did the following:
>
>   *   We added assertion into bsd.buildlink3.mk, we verified that on that scripts the links are properly created

So just after the creation, the process that did the creation was able
to see the symlink, and it was ok.

>   *   At the same time we traced what the build process sees, and these links where not visible to that process.

You said "same time", but I think you mean a compiler that was started
after the previous verification had ended (but in the same build).

>   *   We tried forcing sync, but sync on mac is not guarantied

That's remarkable, and given Unix heritage (4th and Sixth Edition)
unthinkable!  FWIW, my 10.13 man pages don't say this.  But, sync isn't
the point; that is about getting filesystem changes to stable storage,
and your problem is about other processes.

>   *   We tried adding delays, just to see if it changes behavior, and it did not help.



>   *   Once the process fails, you can rerun the build command, and then obviously the builds continues and the process is completed successfully.

When you say "rerun the build command", do you mean "bmake package"?

Are you able to reproduce this?  With what probability?

Can you post a diff (for the assertions)?

And a script that does the build some number of times, each time saving
the output from the first run, and if unsuccessful, runs it again
(saving differently)?

Also include what version of pkgsrc, what version of macOS, and what
kind of hardware.


Overall, it is sounding like

  a make process:

    runs a command that makes a symlink
    runs a command that checks the symlink

  later (definitely after the command that makes a symlink has
  finished), either that make process, or some other one:

    runs a command that (eventually) reads a symlink

      that command does not see the symlink


and if this is true, it sounds like a macOS bug.


You might try creating a test case where a symlink is created and then
some other program looks for it.  You might even make the other program
run under a different uid and get signaled to look via udp or
unix-domain socket.  If you can reproduce failure of filesystem
semantics without pkgsrc, then you certainly have found a macOS bug.


Also, you might try:

  bmake buildlink
  sleep 60
  bmake packaeg

and see if that helps.

I wonder if there is some per-CPU caching of fs state going on that is
not right.