[ On Saturday, November 27, 2004 at 23:24:04 (-0800), Jason Thorpe wrote: ]
> Subject: Re: bge(4) (DEGXA-TX) no-go on the AlphaServer ES40
>
> Ok, please try the following patch against -current.  It implements 
> deferred-until-UP creation of DMA maps for the wm driver, and fixes a 
> DMA resource leak in the bge driver.
> 
> Note: The bge driver is horribly structured, and really needs an 
> overhaul, but I did the minimum work on it for now.

Bge0 still runs out of resources, and much more quickly than I expected
-- or perhaps just when ifconfig changes something?

The wm(4) fixes seem to have dropped performance significantly for this
card with an i82542 chip that I'm currently trying to use.  It has no
hardware csum support so it's not fast in the first place, but now it
seems to be 10MB/s slower than it was before.  At first I thought it was
actually much better, but I was thinking of the 1.6.2 results, but with
-current against patches it may actually be worse (still better than
1.6.2, but not great and apparently not because it maxed out on CPU).
See the ttcp results below....

The good news though is that the wm(4) driver doesn't seem to be the
cause of any more crashes!

Tomorrow we'll replace the i82542 card with a newer one (one with an
i82544 at least, maybe even a newer one than that) and try again.

Next I tried the bge0 interface and managed to get a few pings and a
short ttcp through (with rather abysmal throughput), then I tried to
turn on hardware csums and at that point it stopped transmitting or
receiving and when I tried to turn csums off again and it crashed:


NetBSD 2.99.10 (TSUNAMI.MP) #0: Sun Nov 28 15:38:23 EST 2004
        root@woffi.planix.com:/m5/netbsd-current/obj.alpha/sys/arch/alpha/compile/TSUNAMI.MP
AlphaServer ES40, 666MHz, s/n NI94900217
8192 byte page size, 4 processors.
total memory = 16384 MB
(7080 KB reserved for PROM, 16377 MB used by NetBSD)
avail memory = 16088 MB
mainbus0 (root)
cpu0 at mainbus0: ID 0 (primary), 21264A-14
cpu0: Architecture extensions: 307<PAT,MVI,CIX,FIX,BWX>
cpu1 at mainbus0: ID 1, 21264A-14
cpu1: Architecture extensions: 307<PAT,MVI,CIX,FIX,BWX>
cpu2 at mainbus0: ID 2, 21264A-14
cpu2: Architecture extensions: 307<PAT,MVI,CIX,FIX,BWX>
cpu3 at mainbus0: ID 3, 21264A-14
cpu3: Architecture extensions: 307<PAT,MVI,CIX,FIX,BWX>
tsc0 at mainbus0: 21272 Core Logic Chipset, Cchip rev 0
tsc0: 8 Dchips, 2 memory buses of 32 bytes
tsc0: arrays present: 4096MB (split), 4096MB (split), 4096MB (split), 4096MB (split), Dchip 0 rev 1
tsp0 at tsc0
pci0 at tsp0 bus 0
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
vga0 at pci0 dev 1 function 0: ATI Technologies 3D Rage II+ (rev. 0x9a)
wsdisplay0 at vga0 kbdmux 1
wsmux1: connecting to wsdisplay0
ahc0 at pci0 dev 2 function 0: Adaptec 3960D Ultra160 SCSI adapter
ahc0: interrupting at dec 6600 irq 12
ahc0: aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
scsibus0 at ahc0: 16 targets, 8 luns per target
ahc1 at pci0 dev 2 function 1: Adaptec 3960D Ultra160 SCSI adapter
ahc1: interrupting at dec 6600 irq 13
ahc1: aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
scsibus1 at ahc1: 16 targets, 8 luns per target
isp0 at pci0 dev 3 function 0: QLogic Dual Port FC-AL and 2Gbps Fabric HBA
isp0: interrupting at dec 6600 irq 16
isp0: bad execution throttle of 0- using 16
scsibus2 at isp0: 256 targets, 8 luns per target
tlp0 at pci0 dev 4 function 0: DECchip 21143 Ethernet, pass 3.0
tlp0: interrupting at dec 6600 irq 20
tlp0: DEC DE500-BA, Ethernet address 08:00:2b:c4:b5:26
tlp0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sio0 at pci0 dev 7 function 0: Acer Labs M1543 PCI-ISA Bridge (rev. 0xc3)
Acer Labs M5229 UDMA IDE Controller (IDE mass storage, interface 0xfa, revision 0xc1) at pci0 dev 15 function 0 not configured
Acer Labs M5237 USB 1.1 Host Controller (USB serial bus, interface 0x10, revision 0x03) at pci0 dev 19 function 0 not configured
isa0 at sio0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com0: console
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
pckbc0 at isa0 port 0x60-0x64
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
isabeep0 at pcppi0
fdc0 at isa0 port 0x3f0-0x3f7 irq 6 drq 2
mcclock0 at isa0 port 0x70-0x71: mc146818 or compatible
tsp1 at tsc0
pci1 at tsp1 bus 0
pci1: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
wm0 at pci1 dev 1 function 0: Intel i82542 1000BASE-X Ethernet, rev. 3
wm0: interrupting at dec 6600 irq 24
wm0: Ethernet address 00:d0:b7:82:33:b0
wm0: 1000baseSX, 1000baseSX-FDX, auto
isp1 at pci1 dev 2 function 0: QLogic Dual Port FC-AL and 2Gbps Fabric HBA
isp1: interrupting at dec 6600 irq 28
isp1: bad execution throttle of 0- using 16
scsibus3 at isp1: 256 targets, 8 luns per target
esiop0 at pci1 dev 4 function 0: Symbios Logic 53c895 (ultra2-wide scsi)
esiop0: using on-board RAM
esiop0: interrupting at dec 6600 irq 36
scsibus4 at esiop0: 16 targets, 8 luns per target
bge0 at pci1 dev 6 function 0: Broadcom BCM5703X Gigabit Ethernet
bge0: interrupting at dec 6600 irq 44
bge0: ASIC BCM5703 A2 (0x1002), Ethernet address 00:08:02:91:89:ae
brgphy0 at bge0 phy 1: BCM5703 1000BASE-T media interface, rev. 2
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
fd0 at fdc0 drive 0: 1.44MB, 80 cyl, 2 head, 18 sec
Kernelized RAIDframe activated
scsibus0: waiting 2 seconds for devices to settle...
scsibus1: waiting 2 seconds for devices to settle...
scsibus2: waiting 2 seconds for devices to settle...
scsibus3: waiting 2 seconds for devices to settle...
scsibus4: waiting 2 seconds for devices to settle...
cd0 at scsibus0 target 4 lun 0: <TOSHIBA, CD-ROM XM-5701TA, 0557> cdrom removable
cd0: sync (100.00ns offset 8), 8-bit (10.000MB/s) transfers
sd0 at scsibus1 target 0 lun 0: <COMPAQ, BF01864663, 3B07> disk fixed
sd0: 17365 MB, 7001 cyl, 20 head, 254 sec, 512 bytes/sect x 35565080 sectors
sd0: sync (25.00ns offset 63), 16-bit (80.000MB/s) transfers, tagged queueing
sd1 at scsibus1 target 1 lun 0: <COMPAQ, BF01864663, 3B07> disk fixed
sd1: 17365 MB, 7001 cyl, 20 head, 254 sec, 512 bytes/sect x 35565080 sectors
sd1: sync (25.00ns offset 63), 16-bit (80.000MB/s) transfers, tagged queueing
sd2 at scsibus1 target 2 lun 0: <COMPAQ, BF03685A35, HPB7> disk fixed
sd2: 34732 MB, 31310 cyl, 4 head, 567 sec, 512 bytes/sect x 71132000 sectors
sd2: sync (12.50ns offset 63), 16-bit (160.000MB/s) transfers, tagged queueing
sd3 at scsibus1 target 3 lun 0: <COMPAQ, BF03685A35, HPB7> disk fixed
sd3: 34732 MB, 31310 cyl, 4 head, 567 sec, 512 bytes/sect x 71132000 sectors
sd3: sync (12.50ns offset 63), 16-bit (160.000MB/s) transfers, tagged queueing
sd4 at scsibus1 target 4 lun 0: <COMPAQ, BF03685A35, HPB7> disk fixed
sd4: 34732 MB, 31310 cyl, 4 head, 567 sec, 512 bytes/sect x 71132000 sectors
sd4: sync (12.50ns offset 63), 16-bit (160.000MB/s) transfers, tagged queueing
sd5 at scsibus2 target 1 lun 0: <APPLE, Xserve RAID, 1.21> disk fixed
sd5: 1402 GB, 179526 cyl, 128 head, 128 sec, 512 bytes/sect x 2941353984 sectors
sd6 at scsibus3 target 1 lun 0: <APPLE, Xserve RAID, 1.21> disk fixed
sd6: 1402 GB, 179526 cyl, 128 head, 128 sec, 512 bytes/sect x 2941353984 sectors
sd2: no disk label
sd3: no disk label
sd4: no disk label
root on sd1a dumps on sd1b
root file system type: ffs
/etc/rc.conf is not configured.  Multiuser boot aborted.

  N O T I C E :  Please do not use the console except to run shutdown!

We recommend creating a non-root account and using su(1) for root access.
Terminal type is wsvt25m.
We recommend creating a non-root account and using su(1) for root access.
[console]<@> # fsck /
** /dev/rsd1a
** File system is clean; not checking
[console]<@> # ifconfig wm0 inet 10.20.20.2 netmask 255.255.255.0 up
[console]<@> # ifconfig wm0
wm0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        address: 00:d0:b7:82:33:b0
        media: Ethernet autoselect (1000baseSX full-duplex)
        status: active
        inet 10.20.20.2 netmask 0xffffff00 broadcast 10.20.20.255
[console]<@> # ttcp -v -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 10.20.20.1
ttcp-r: 819200000 bytes in 28.82 real seconds = 27758.91 KB/sec +++
ttcp-r: 819200000 bytes in 28.81 CPU seconds = 27771.80 KB/cpu sec
ttcp-r: 100004 I/O calls, msec/call = 0.30, calls/sec = 3470.00
ttcp-r: 0.0user 28.7sys 0:28real 99% 0i+0d 0maxrss 0+1pf 3+319csw
ttcp-r: buffer address 0x20050000
[console]<@> # ttcp -v -t -s -n 10000 10.20.20.1 
ttcp-t: buflen=8192, nbuf=10000, align=16384/0, port=5001  tcp  -> 10.20.20.1
ttcp-t: socket
ttcp-t: connect
ttcp-t: 81920000 bytes in 3.54 real seconds = 22584.18 KB/sec +++
ttcp-t: 81920000 bytes in 3.54 CPU seconds = 22590.88 KB/cpu sec
ttcp-t: 10000 I/O calls, msec/call = 0.36, calls/sec = 2823.02
ttcp-t: 0.0user 3.5sys 0:03real 99% 0i+0d 0maxrss 0+10001pf 3+38csw
ttcp-t: buffer address 0x20050000
[console]<@> # ifconfig bge0
bge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=0<>
        address: 00:08:02:91:89:ae
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
[console]<@> # ifconfig bge0 inet 10.10.10.2 netmask 255.255.255.0 up
bge0: pcistate failed to revert
bge0: pcistate failed to revert
[console]<@> # ping 10.10.10.1
PING 10.10.10.1 (10.10.10.1): 48 data bytes
64 bytes from 10.10.10.1: icmp_seq=0 ttl=64 time=0.778 ms
64 bytes from 10.10.10.1: icmp_seq=1 ttl=64 time=1.299 ms
64 bytes from 10.10.10.1: icmp_seq=2 ttl=64 time=1.345 ms
64 bytes from 10.10.10.1: icmp_seq=3 ttl=64 time=1.448 ms
64 bytes from 10.10.10.1: icmp_seq=4 ttl=64 time=1.392 ms
64 bytes from 10.10.10.1: icmp_seq=5 ttl=64 time=1.331 ms
64 bytes from 10.10.10.1: icmp_seq=6 ttl=64 time=1.387 ms
64 bytes from 10.10.10.1: icmp_seq=7 ttl=64 time=1.338 ms
64 bytes from 10.10.10.1: icmp_seq=8 ttl=64 time=1.261 ms
64 bytes from 10.10.10.1: icmp_seq=9 ttl=64 time=1.346 ms
64 bytes from 10.10.10.1: icmp_seq=10 ttl=64 time=1.278 ms
64 bytes from 10.10.10.1: icmp_seq=11 ttl=64 time=1.373 ms
64 bytes from 10.10.10.1: icmp_seq=12 ttl=64 time=1.321 ms
64 bytes from 10.10.10.1: icmp_seq=13 ttl=64 time=1.262 ms
^C
----10.10.10.1 PING Statistics----
14 packets transmitted, 14 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.778/1.297/1.448/0.158 ms
[console]<@> # ttcp -v -t -s -n 10000 10.10.10.1    
ttcp-t: buflen=8192, nbuf=10000, align=16384/0, port=5001  tcp  -> 10.10.10.1
ttcp-t: socket
ttcp-t: connect
ttcp-t: 81920000 bytes in 9.77 real seconds = 8192.15 KB/sec +++
ttcp-t: 81920000 bytes in 4.60 CPU seconds = 17393.70 KB/cpu sec
ttcp-t: 10000 I/O calls, msec/call = 1.00, calls/sec = 1024.02
ttcp-t: 0.0user 4.5sys 0:09real 46% 0i+0d 0maxrss 0+10001pf 0+107csw
ttcp-t: buffer address 0x20050000
[console]<@> # ifconfig bge0 ip4csum tcp4csum udp4csum
bge0: pcistate failed to revert
[console]<@> # ifconfig bge0                          
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        address: 00:08:02:91:89:ae
        media: Ethernet autoselect (1000baseT full-duplex,master)
        status: active
        inet 10.10.10.2 netmask 0xffffff00 broadcast 10.10.10.255
[console]<@> # ping 10.10.10.1                        
PING 10.10.10.1 (10.10.10.1): 48 data bytes
^C
----10.10.10.1 PING Statistics----
12 packets transmitted, 0 packets received, 100.0% packet loss
[console]<@> # ifconfig bge0 -ip4csum -tcp4csum -udp4csum
bge0: pcistate failed to revert
bge0: unable to load std Rx DMA map (idx 0), error = 35
panic: bge_newbuf_std
Stopped in pid 52.1 (ifconfig) at       netbsd:cpu_Debugger+0x4:        ret     zero,(ra)
db{0}> trace
cpu_Debugger() at netbsd:cpu_Debugger+0x4
panic() at netbsd:panic+0x1f8
bge_newbuf_std() at netbsd:bge_newbuf_std+0x3fc
bge_init_rx_ring_std() at netbsd:bge_init_rx_ring_std+0x54
bge_init() at netbsd:bge_init+0x27c
bge_ioctl() at netbsd:bge_ioctl+0x198
ifioctl() at netbsd:ifioctl+0x694
soo_ioctl() at netbsd:soo_ioctl+0xf8
sys_ioctl() at netbsd:sys_ioctl+0x134
syscall_plain() at netbsd:syscall_plain+0xdc
XentSys() at netbsd:XentSys+0x60
--- syscall (54) ---
--- user mode ---
db{0}> 




FYI here's the same TTCP test on wm0 to the test receiver from 1.6.2_STABLE:

[console]<@> # ttcp -t -s -v -n 10000 10.20.20.1
ttcp-t: buflen=8192, nbuf=10000, align=16384/0, port=5001  tcp  -> 10.20.20.1
ttcp-t: socket
ttcp-t: connect
ttcp-t: 81920000 bytes in 4.99 real seconds = 16042.37 KB/sec +++
ttcp-t: 81920000 bytes in 4.88 CPU seconds = 16377.53 KB/cpu sec
ttcp-t: 10000 I/O calls, msec/call = 0.51, calls/sec = 2005.30
ttcp-t: 0.2user 4.6sys 0:04real 97% 0i+0d 0maxrss 0+1pf 6+54csw
ttcp-t: buffer address 0x20050000


and with -current from before your if_wm.c patches:

[console]<@> # ttcp -s -v -t -n 50000 10.20.20.1
ttcp-t: buflen=8192, nbuf=50000, align=16384/0, port=5001  tcp  -> 10.20.20.1
ttcp-t: socket
ttcp-t: connect
ttcp-t: 409600000 bytes in 12.32 real seconds = 32478.45 KB/sec +++
ttcp-t: 409600000 bytes in 8.46 CPU seconds = 47290.62 KB/cpu sec
ttcp-t: 50000 I/O calls, msec/call = 0.25, calls/sec = 4059.81
ttcp-t: 0.0user 8.4sys 0:12real 68% 0i+0d 0maxrss 0+50001pf 3+146csw
ttcp-t: buffer address 0x20050000


With your patches wm(4) is still much faster than in 1.6.2, but I was
assuming the 1.6 -> current improvement had more to do with the software
checksumming code than anything else.


-- 
						Greg A. Woods

+1 416 218-0098                  VE3TCP            RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>          Secrets of the Weird <woods@weird.com>