Very slow filesystem

[Stephen Borrill <netbsd%precedence.co.uk@localhost>]

I've a machine which has a filesysmte that is extremely slow to create 
dirs and files. Look at the time to run mkdir below:

backup 1# time mkdir /data/backup/previous4
0.000u 0.131s 1:31.95 0.1%      0+0k 7912+5io 0pf+0w
backup 2# df -hi
Filesystem        Size       Used      Avail %Cap    iUsed   iAvail %iCap 
Mounted on
/dev/sd0a         5.9G       1.6G       4.0G  28%    81236   689066  10% /
/dev/raid0d       7.2T       6.0T       853G  87%  7004623 474545967   1% 
/data
tmpfs             2.2G       8.0K       2.2G   0%       14 16612892   0% 
/tmp
kernfs            1.0K       1.0K         0B 100%        0        0   0% 
/kern
ptyfs             1.0K       1.0K         0B 100%        0        0   0% 
/dev/pts
procfs            4.0K       4.0K         0B 100%       49      995   4% 
/proc
backup 3# time rmdir /data/backup/previous4
0.000u 0.001s 0:00.06 0.0%      0+0k 1+6io 0pf+0w
backup 4# uname -a
NetBSD backup 6.1_STABLE NetBSD 6.1_STABLE (MONOLITHIC) #5: Wed Oct 23 
15:10:22 BST 2013 
root%builder.internal.precedence.co.uk@localhost:/usr/obj/6.0/i386/sys/arch/i386/compile/MONOLITHIC 
i386

Filesystem is completely idle. It's on a 5-disk RAID 5 RAIDframe array. 
fsck reports no problems. WAPBL is not enabled.

Creating a new top-level directory in the mount is OK, but it is very 
slow in any subdir of the backup directory (which contains probably 95% 
of the data).

bonnie suggests that underlying performance is pretty good:

-------Sequential Output-------- ---Sequential Input-- --Random--
-Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
64835 71.8 49498 25.3 16177  8.5 58207 76.3 124726 36.9 148.8  1.6

--
Stephen