kern/38485: possible optimization for bind

[ad%netbsd.org@localhost]

>Number:         38485
>Category:       kern
>Synopsis:       possible optimization for bind
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    kern-bug-people
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Wed Apr 23 07:30:00 +0000 2008
>Originator:     Andrew Doran
>Release:        4.99.60 + fine grained socket locking
>Organization:
The NetBSD Project
>Environment:
n/a
>Description:
Multi-cpu, authoritative bind servers seem have one worker thread per-CPU,
all communicating with the outside world over a single, shared UDP socket.

bind seems to make heavy use of sendmsg/recvmsg. In soreceive/sosend, we
acquire the "long term" socket buffer lock across the entire operation,
and make use of the short term socket lock for atomic access to data
structures. As one CPU drops the socket lock, it's possible that other
CPUs will find the socket buffer locked and sleep, which would lead to a 
situation where the worker threads constantly thrash on the long term 
socket buffer locks.

The short term lock is dropped for operations like uiomove that could
hang waiting for memory to become available. In practice those operations
hardly ever hang.

>How-To-Repeat:
Code inspection, I haven't tried it yet.

>Fix:
1. Make the socket buffer locks behave like adaptive mutexes.

2. If sendmsg/recvmsg are sending only short messages (MSIZE or smaller?)     
then prepackage the data into mbufs rather than relying on the uiomove()
further down the chain.

3. ???