Re: Severe netbsd-6 NFS server-side performance issues

[buhrow%lothlorien.nfbcal.org@localhost (Brian Buhrow)]

Hello.  I went back and re-read your original message.  It would be
interesting to know what those nfsd's are doing when they're in d state.
What does ps -l show?   Is it possible to build a kernel with debugging
symbols and then get things in a plugged up state, then drop into the ddb
debugger and get a crash dump?  If you can do that, then gdb can be used on
the crash file after the fact to determine what each process is doing at
the time of the crash.  I still think it has something to do with the raid
controller even if it's not obvious.  Is anyone else using the crazy
version of firmware you picked up?
        Also, what happens if you run two or more bonnie processes
simultaneously?  That ought to stress the raid out and help demonstrate the
problem.
-Brian

On May 30, 10:31pm, Hauke Fath wrote:
} Subject: Re: Severe netbsd-6 NFS server-side performance issues
} At 12:45 Uhr -0700 30.5.2012, Brian Buhrow wrote:
} >Hmm.  A panic a week?  I wouldn't call that stable at all.
} 
} That's relative...  ;)
} 
} >However, one issue at a time.
} >     Does the raid controller have some sort of battery backed up cache?
} 
} Yes.
} 
} >Which driver supports these raid controllers?  mfi(4)?
} 
} amr0 at pci4 dev 0 function 0: AMI RAID <MegaRAID SCSI 320-4X>
} amr0: interrupting at ioapic2 pin 4
} amr0: firmware 421D, BIOS H434, 256MB RAM
} ld0 at amr0 unit 0: RAID 5, optimal
} ld0: 1675 GB, 218759 cyl, 255 head, 63 sec, 512 bytes/sect x 3514368000 
sectors
} 
} -- I had fun upgrading the firmware (the original version would install the
} RAID, but not boot), since the controller turned out to be a Fujitsu OEM,
} although it said LSI all over the board, and LSI refused to support it. In
} the end, I pried out a totally undocumented firmware from an iso image off
} an obscure Fujitsu ftp server. It worked, and even fixed the immediate
} problem.
} 
} >     I'm not sure of the problem, but it sounds like some sort of write
} >back cacheing switch got flipped in the raid controller, and now writes,
} >with interleaving reads, are really slow.  I'm assuming you've looked at
} >the BIOS of the raid controller to see if there are any settings which
} >might have to do with caching?
} 
} Yes, and they are as I wanted them, battery present, cache set to
} write-back. I ran bonnie++ on the array last week to check whether the
} dismal performance is disk related. From the results I got I'd say it is
} not.
} 
} >     Is it possible to revert to the NetBSD-5.x system and see if your
} >performance goes back  to where it was  after the raid upgrade but before
} >the OS upgrade?
} 
} Tricky. The installation is netbsd-6, and the disk array is well-filled, so
} I'd have to attach a sata disk for the netbsd-5 system, plus rebuild the
} packages installed. Might be worth a try, though, although as I said, the
} netbsd-5 stability was not satisfactory.
} 
} >     If you export a non-raid disk, what kind of nfs performance do you get
} >on the system when writing to that disk?  (That will help you figure out if
} >it's a raid problem or an OS problem.)
} 
} Hm - I could try that quickly tomorrow. Might well give some information.
} 
} OTOH, the bonnie++ run appears to have cleared the disk subsystem.
} 
} >     I realize that I'm more questions than answers, but I think the
} >question you're trying to answer is, what changed?  The answer is, I think,
} >something about the raid system.
} 
} Nothing really changed, that's what baffles me. The machine rebooted after
} a panic at some point, and came up with bad performance. I had updated the
} kernel in that context, but as I mentioned, the old kernel gave the same
} result.
} 
} At this point, I am very much banging my head against a wall, so questions,
} by forcing me to re-think the situation from the ground up, are quite
} helpful.
} 
}       hauke
} 
} 
} 
} 
} 
} --
} "It's never straight up and down"     (DEVO)
} 
} 
>-- End of excerpt from Hauke Fath