Re: RFC: MSI/MSI-X implementation

[Kengo NAKAHARA <k-nakahara%iij.ad.jp@localhost>]

Hi,

(2014/11/14 4:59), David Young wrote:
> On Thu, Nov 13, 2014 at 12:41:38PM +0900, Kengo NAKAHARA wrote:
> > (2014/11/13 11:54), David Young wrote:
> > > On Fri, Nov 07, 2014 at 04:41:55PM +0900, Kengo NAKAHARA wrote:
> > > > Could you comment the specification and implementation?
> > >
> > > The user should not be on the hook to set processor affinity for the
> > > interrupts.  That is more properly the responsibility of the designer
> > > and OS.
> >
> > I wrote unclear explanation..., so please let me redescribe.
> >
> > This MSI/MSI-X API *design* is independent from processor affinity.
> > The device dirvers can use MSI/MSI-X and processor affinity
> > independently of each other. In other words, legacy interrupts and
> > INTx interrupts can use processor affinity still. Furthermore,
> > MSI/MSI-X may or may not use processor affinity.
>
> MSI/MSI-X is not half as useful as it ought to be if a driver's author
> cannot spread interrupt workload across the available CPUs.  If you
> don't mind, please share your processor affinity proposal and show how
> it works with interrupts.

The device dirver's author can use processor affinity kernel API,
please see:
    http://mail-index.netbsd.org/tech-kern/2014/08/27/msg017584.html
# Above intrctl(9) usage is old. Here is current usage:
#     http://mail-index.netbsd.org/tech-kern/2014/09/12/msg017653.html

Below example is userland usage, but the working is the same as using
kernel API. Please see:
    http://mail-index.netbsd.org/tech-kern/2014/08/20/msg017522.html
    http://mail-index.netbsd.org/tech-kern/2014/11/07/msg017897.html

In msg017897.html example, if_vmx driver does not use intr_distribute()
API, so all of "vmx0"'s interrupts(msix0 vec [0-4]) are set affinity to
default cpu(cpu0). But, they can be moved other cpus by using intrctl(9)
after kernel boot completed. When intrctl(9) is completed, the interrupts
of "vmx0", which use multiqueue, are spread to 2 cpus. In particular
     - "msix0 vec 0"(interrupt for 1st TX queue) is set affinity to cpu0
     - "msix0 vec 1"(interrupt for 2nd TX queue) is set affinity to cpu0
     - "msix0 vec 2"(interrupt for 1st RX queue) is set affinity to cpu1
     - "msix0 vec 3"(interrupt for 2nd RX queue) is set affinity to cpu1
     - "msix0 vec 4"(interrupt for link state changing) is set affinity to cpu0

This affinity is just one example. It may be better to set affinity 1st
RX queue to cpu0 and set affinity to 2nd RX queue to cpu1. I think the
administrator decides how to set affinity. The administrator who is not
interested in performance may use the system with default affinity.

Thanks,

--
//////////////////////////////////////////////////////////////////////
Internet Initiative Japan Inc.

Device Engineering Section,
Core Product Development Department,
Product Division,
Technology Unit

Kengo NAKAHARA <k-nakahara%iij.ad.jp@localhost>