Re: bnx driver in 5.99.7 panics under xen 3.3.1

[Mike Bowie <mbowie%buzmo.com@localhost>]

Manuel Bouyer wrote:
> On Wed, Jan 28, 2009 at 04:25:39PM -0800, Mike Bowie wrote:
>> Mike Bowie wrote:
>>> Manuel Bouyer wrote:
>>>> On Tue, Jan 27, 2009 at 01:30:28PM -0800, Mike Bowie wrote:
>>>>  
>>>>> [...]
>>>>> bnx0 at pci8 dev 0 function 0: Broadcom NetXtreme II BCM5708 1000Base-T
>>>>> ioapic1: int5
>>>>> 0x1a9a8<vector=0xa8,delmode=0x1,logical,actlo,level,masked,dest=0x0>
>>>>> 0xff000000<target=0xff>
>>>>> bnx0: Ethernet address 00:22:19:53:d8:a3
>>>>> xen_alloc_contig: XENMEM_decrease_reservation failed!
>>>>> bnx0: Could not create Tx mbuf 0 DMA map!
>>>>>     
>>>> This is more serious than just a bnx issue. I can't see how a
>>>> XENMEM_decrease_reservation can fail, unless it gets passed nonexistent
>>>> pages. Can you try booting xen-debug.gz instead of xen.gz, and see
>>>> if xen has something to say about this XENMEM_decrease_reservation
>>>> operation (should appear just before the NetBSD kernel reports that
>>>> it failed).
>>>>
>>>>   
>>> Hrm... in a bizarre twist of fate, xen-debug.gz actually manages to get
>>> past the bnx init.  Here's the serial dump, not sure why it stops where
>>> it does, but I won't be able to look at that until tonight.
>>>
>> Just to confirm, the VGA console is blank, so the serial output is all
>> there is.  <ctrl>+aaa does switch to Xen, but dom0 is unresponsive
>> following the output I posted earlier.  (Besides echoing any input.)
> 
> After swicthing to xen you could try 'q' and see if the dom0 prints
> anything on its console. 'd' may also be usefull if you can match the
> addresses printed with the netbsd.gdb kernel using gdb.
> 
> I remember having strange issues with a 4GB ram dell 2950 and xen 
> (3.1.x/i386).
> The dom0 kernel would segfault unless I reduce the amount of ram seen by
> the xen kernel using mem=3072M on the xen.gz boot line (xen.gz did see
> something like 3200M as it's a 32bit kernel).
> A supermicro server also with 4GB ram didn't have troubles with the same
> xen and dom0 kernels.
> 
> My theroy at that time was that the Dell BIOS did provide either a bogus
> memory segment to xen, or that xen misinterpreted the BIOS's memory map.
> This was probably the last memory segment as the dom0 kernel is loaded
> in the top of the RAM. You can try reducing a bit the ram availble to xen
> like I did.
> 

Well well...

It would appear that adding the mem restriction to Xen did the trick.  I
didn't go so far as to compare the addresses as I'm a bit short on time
this week, but perhaps early next week I'll take a look.

The reason the debug kernel didn't boot through was actually due to my
haste in setting up /boot.cfg; on account of Xen and NetBSD differing on
their com port numbering... the following line has resulted in a
successful boot.

menu=Boot Xen with 512MB for dom0 (serial):load /netbsd-XEN3_DOM0
console=com1;multiboot /usr/pkg/xen3-kernel/xen.gz mem=3072M
dom0_mem=512M console=com2 com2=9600,8n1

I'm going to stress the box with a couple of domU's building stuff over
the next few days and see how it plays out.  I may even reduce dom0's
RAM alloc a bit, given that I've just sacrificed 1GB into oblivion.  (I
may try other values for the cut-off too, for the sake of interest.)

One thing worth noting is that this hardware required a certain DIMM
configuration with dual processors... which I haven't seen before.
Granted, I don't stay all that current with the latest and greatest, but
the configuration in this case required at least four physical DIMM's.

This box isn't due online until the 1st of March, so I have some time to
experiment and test a bit.

My sincere thanks for the feedback... I'll post my findings on RAM
changes etc.

Cheers,

Mike.