Re: CVS commit: src/lib/libc/stdlib

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Thu, 29 Sep 2022 08:18:49 -0400
    From:        Christos Zoulas <christos%zoulas.com@localhost>
    Message-ID:  <C6F50768-0E4F-478F-A3FE-5643A70EA6C9%zoulas.com@localhost>

  | Yes, I had forgotten about the need to do this explicitly...

  | > On Sep 28, 2022, at 10:23 PM, Robert Elz <kre%munnari.OZ.AU@localhost> wrote:
  | > 
  | > Apologies, I did not read the code closely enough, there must be a bug
  | > in the way the clone file descriptor is created.

I think I know the cause now, and it isn't anything specific to /dev/ptmx
it would affect all cloning devices.

In kern/vfs_syscalls.c do_open() we see the following:

        if (vp == NULL) {
                fd_abort(p, fp, indx);
                error = fd_dupopen(dupfd, dupfd_move, flags, &indx);
                if (error)
                        return error;
                *fd = indx;   
        } else {
                error = open_setfp(l, fp, vp, indx, flags);
                if (error)
                        return error;
                VOP_UNLOCK(vp);
                *fd = indx;
                fd_affix(p, fp, indx);
        }

The vp==NULL case is where cloning devices are handled.   fd_dupopen()
arranges top make the duplicate happen.

It attempts to handle O_CLOEXEC thus:

	error = fd_dup(fp, 0, newp, ff->ff_exclose);

where ff_exclose is the close_on_exec bit for the old file descriptor
(the one being duplicated) which is just fine for other calls of fd_dupopen().

The vp!=NULL case above is for "normal" opens, and open_setfp() is what
handles O_CLOEXEC - that's what (eventually) makes ff_exclose be true
(it also handles O_EXLOCK and O_SHLOCK.

None of that is being done in the cloning device open case, so those 3
flags simply cannot work, as the code is written currently, for these
devices.

I'm not about to go playing in this very fiddly piece of code, but I'm
kind of hoping that dholland@ might know the right magic sequence to make
this issue go away.

It probably means a call to open_setfp() with some parameters or other,
somewhere in the vp==NULL side of that if statement, though, just what
I have not attempted to work out.

kre