Re: updating COMPAT_LINUX for linux 2.6.x support.

[Chuck Silvers <chuq%chuq.com@localhost>]

On Mon, May 03, 2010 at 05:06:12AM +0100, Mindaugas Rasiukevicius wrote:
> Hello,
> 
> Chuck Silvers <chuq%chuq.com@localhost> wrote:
> > hi folks,
> > 
> > I've started looking at what it would take to update COMPAT_LINUX
> > to allow changing the emulated linux kernel version to 2.6.x.
> 
> Great!
> 
> > (2) what linux kernel version do we want to claim to be in uname?
> >     it would be good for this to be at least as recent as the
> >     kernel matching what we choose in (1).  but it would good to avoid
> >     going too wild with this, to keep the amount of new emul code
> >     that's needed to a minimum.
> 
> Well, we should probably update the list of emulated syscalls and see how
> many of them we can properly support.  Depending on that, figure out a
> highest version to claim.

sure, I was just wondering if anyone had done this already.

I looked through the list, and found that there are actually a couple dozen
that we already claimed support for (ie. that were present in linux 2.4.18)
that we don't implement, so this isn't a hard requirement.
looking at all the ones added since then, other than the threading-related
ones, none of them seem likely to be widely used.  so it looks like
it doesn't matter a whole lot what version we claim to be.


> >     I tried following the suggestion in PR 37437 (moving the ps_sigignore
> >     and ps_sigcatch fields from struct sigctx to struct sigacts, which is
> >     shared between linux threads), and I moved p_sigpend from struct proc
> >     to struct sigacts as well.  this, together with other fixes allows
> >     all of the netbsd libpthread regression tests to pass (except the one
> >     that also fails natively on linux), and gives much improved results
> >     on the LTP/posixtest tests.  I put my current diff (still larded with
> >     debug stuff and with linux stuff hardcoded instead of called through
> >     new emul callbacks) in:
> >     ftp://ftp.netbsd.org/pub/NetBSD/misc/chs/linux/diff.linux-nptl.4
> 
> This way adds locking overhead and extra complexity we do not really want
> in the native signalling code.  As Andrew suggested in the PR, perhaps a
> better way would be re-using a proc_t::p_lock, with reference counting on
> the lock, i.e. using mutex_obj_hold(9).  But see below..

andrew actually made those changes, the checkin log is later in the same PR.
the current linux emulation code relies on that plus sharing some of the
signal state.  my earlier changes built on top of that by sharing even more
of the signal state.


> >     before continuing down this path, I was wondering if it would be
> >     possible to move from the current way of supporting linux threads
> >     (each linux thread is a separate netbsd process, with extra sharing
> >     happening behind the scenes) to the native netbsd way (1 process with
> >     multiple LWPs)?  the only problem I can think of is that something
> >     might assume that linux thread IDs are globally unique, or that the
> >     thread ID of the original thread is the same as the process ID.
> >     but that seems a bit unlikely, since I don't see any place that the
> >     linux thread ID is used (thought I could easily have missed that).
> >     does anyone know of any reason why this wouuldn't work?
> 
> Right, I think supporting Linux threads (of NPTL world) as native LWPs,
> instead of processes is a major step we need to do.  Apart from potential
> performance advantages, that should avoid p_sigctx sharing complexity,
> should not it?

yes, that's why I proposed it.  :-)


> AFAIK, Linux thread IDs (TIDs) are system-wide unique.  If so, we need to
> ensure that.  Any reason why PID allocator cannot be re-used for that, or
> perhaps even simpler way to fake it with e.g. lwp_t * pointer value?

I was hoping that the TID namespace didn't really need to be the same as
the PID namespace, but further investigation showed that it does.
changing the PID allocator to allow associating multiple PIDs with the
the same process turned out to be easy, though.


> Another thing - we need a native thread-local storage (TLS).  I suppose
> we will need some support for COMPAT_LINUX too.
> 
> >     does anyone know of other work that needs to be done on COMPAT_LINUX
> >     before we can usefully report ourselves as 2.6.x?
> 
> I had unfinished patch somewhere to translate asynchronous I/O functionality.
> Can also improve support in few more areas, like scheduling and affinity.

sure, there will always be more to do here, it's a moving target.


> What is the target, 6.0 release? :)

my goal is ASAP, and I have a redone version almost finished.
I'll post more about that shortly.

-Chuck