I had a thought about optimizing mbufs.  Allow each mbuf chain only one
2k or 4k buffer, whichever buffer is big enough to fit every packet.
All mbufs in the chain point their m_data into this buffer.  Remove all
chain cleverness for prepending/concatenating/trimming mbufs: just
memmove or memcpy into the buffer.  For every MGET/MGETHDR caller, keep
an "exemplar" mbuf chain.  Whenever an mbuf operation demands a memmove,
rearrange the exemplar chain so that the memmove can be avoided next time.
For a trivial example, say we have this mbuf that originates with a
socket, X:

+----------------------------------------------+
| m_data = buffer | m_len = 250 | m_origin = X |
+----------------------------------------------+
 |
 |
 |
 v
+-----------------------------------------------------+
|buffer                                               |
+-----------------------------------------------------+

wi(4) calls m_adj/m_prepend in order to replace the Ethernet II header
with 802.11+802.2 headers.  This requires an memmove.  For some-socket,
we update the exemplar:

+--------------------------------------------------+
| m_data = buffer + len(802.11+802.2) - len(ethii) |-------+
+--------------------------------------------------+       |
| m_len = 4k - len(802.11+802.2) + len(ethii)      |       |
+--------------------------------------------------+       |
| m_origin = X                                     |       |
+--------------------------------------------------+       |
                                                           |
             +---------------------------------------------+
             |
             |
             v
+------------+----------------------------------------+
|buffer                                               |
+------------+----------------------------------------+

Next time socket X demands a new mbuf, it is copied from an exemplar
that helps avoid the memmove.

You can also optimize the exemplar for both alignment (device drivers
or loaning may demand word/page boundary alignment) and for mid-packet
insertion/deletion (I do not recall ever seeing that).  With slight
changes to the mbuf API, I think an exemplar can use external storage,
also.

Dave

-- 
David Young             OJC Technologies
dyoung@ojctech.com      Urbana, IL * (217) 278-3933