software interrupts & scheduling oddities

[David Young <dyoung%pobox.com@localhost>]

Maybe this has happened to you: you tune your NetBSD router for fastest
packet-forwarding speed.  Presented with a peak packet load, your
router does really well for 30 seconds.  Then it reboots because the
user-tickle watchdog timer expires.  OR, your router doesn't reboot but
you cannot change any parameters because the user interface doesn't get
any CPU cycles.  This is the problem that I am dealing with today: while
the system is doing a lot of network processing, the userland doesn't
make any progress at all.  Userland seems to be starved of CPU cycles
because there is a non-stop software-interrupt load caused by the high
level of network traffic.  At least on i386, if there is any software
interrupt pending, then it will run before any user process gets a
timeslice.  So if the softint rate is really high, then userland will
scarcely ever run.  Or that is my current understanding.  Is it incorrect?

Ordinarily, under high packet load, processes that I need to stay
interactive, such as my shell, freeze up after the network load reaches
a certain level.  If I change the scheduling (class, priority) for my
shell to (SCHED_RR, 31), then the shell stays responsive, even though it
still runs at a lower priority than softints.  Ok, so maybe that makes
sense: of all the userland processes, my shell is the only one running
with a real-time priority, so if there are any cycles leftover after the
softints run, my shell is likely to get them.

I thought that maybe, if I run every process at (SCHED_RR, 31) so that
my shell has to share the leftover cycles with every other user process,
then my shell will freeze up again under high packet load.  I set
every user process to (SCHED_RR, 31), though, and the shell remained
responsive.

I'm using the SCHED_M2 scheduler, btw, on a uniprocessor.  SCHED_M2 is
kind of an arbitrary choice.  I haven't tried SCHED_4BSD, yet, but I
will.

I don't really expect for changing any process class/priority to
SCHED_RR/31 to make any difference in the situation I describe, so there
must be something that I am missing about the workings of the scheduler.

One more thing: userland processes get a priority bump when they enter
the kernel.  No problem.  But it seems like a bug that the kernel will
also raise the priority of a low-priority *kernel* thread if it, say,
waits on a condition variable.  I think that happens because cv_wait()
calls cv_enter(..., l) that sets l->l_kpriority, which is only reset by
mi_userret().  Kernel threads never go through mi_userret() so at some
point the kernel will call lwp_eprio() to compute an effective priority:

static inline pri_t
lwp_eprio(lwp_t *l)
{
        pri_t pri;

        pri = l->l_priority;
        if (l->l_kpriority && pri < PRI_KERNEL)
                pri = (pri >> 1) + l->l_kpribase;
        return MAX(l->l_inheritedprio, pri);
}

Since my low-priority kernel thread has lower priority than PRI_KERNEL,
and l_kpriority is set, it gets bumped up.  Perhaps lwp_eprio() should
test for kernel threads (LW_SYSTEM) before elevating the priority?

static inline pri_t
lwp_eprio(lwp_t *l)
{
        pri_t pri;

        pri = l->l_priority;
        if (l->l_kpriority && pri < PRI_KERNEL && (l->l_flag & LW_SYSTEM) == 0)
                pri = (pri >> 1) + l->l_kpribase;
        return MAX(l->l_inheritedprio, pri);
}

Dave

-- 
David Young
dyoung%pobox.com@localhost    Urbana, IL    (217) 721-9981