50% slowdown on 2 processor XEN 4.1.3 vs 1 processor system

[Harry Waddell <waddell%caravaninfotech.com@localhost>]

I just built a new server with 2 x E5-2630 processors and was comparing the 
performance to a nearly identical xen server with 1 x E5-1660 processor and
found that on a per core basis, instead of being about 50% faster ( as one 
would expect from the clock speed given that the architecture is nearly 
identical ), the E5-1660 system is %300 faster, so I ran some benchmarks and 
started looking for a pattern. 

Both systems run NetBSD amd64 6.0-STABLE with xen 4.1.3. The 2
processor system is running only 1/2 as fast with the XEN3_DOM0 kernel
as with the GENERIC kernel, e.g. with simple single threaded benchmarks
like dhrystone and whetstone. ( 22624434.0 vs 10416667.0 dhrystones -- It's 
also clearly slower when compiling, etc... ) The one processor system works 
just fine. 

So far, I've tried: 

1. a netbsd 6.0 release XEN3_DOM0 kernel

2. run the test in a netbsd domU

3. disabled HT and NUMA

4. used xl to create pools based on NUMA nodes and assigned/pinned dom0 to one

5. compiled and installed xen 4.2.0rc4

and nothing seems to influence the disparity in the performance. 

Has anyone else seen similar behavior, or does anyone have any suggestions on 
how to proceed? Removing a cpu is kind of dangerous, so I'd like to avoid that, 
but if there is a good xen dom0 linux live CD, or something similar, I could 
try booting and testing under linux? It was pretty difficult getting netbsd to 
install and boot on my 6TB raid 5 that
I'd hate to blow that away for such a test, but I do have usb keys I could 
install into etc. I assume that if linux dom0's had this issue I would have 
found something about it during my searches, so I'm guessing this is a BSD 
issue, but that's still just an assumption, so if there's an easyish way to 
test that theory, I'll do it. 

Thanks, 

Harry Waddell