Re: troubleshooting I/O performance bottleneck

[Thor Lancelot Simon <tls%rek.tjls.com@localhost>]

On Thu, Apr 30, 2009 at 07:16:07PM -0400, Jed Davis wrote:
> Manuel Bouyer <bouyer%antioche.eu.org@localhost> writes:
> 
> > On Tue, Apr 28, 2009 at 08:13:36AM -0400, Thor Lancelot Simon wrote:
> >> On Tue, Apr 28, 2009 at 01:26:01PM +0200, Manuel Bouyer wrote:
> >> > 
> >> > This is also what I get on my test box. iostat -x shows that the vnd0
> >> > device is getting write requests of 32k (this is expected as the
> >> > xbd interface can't do 64k transfers so MAXPHYS is set to 32k for domUs),
> >> > and writes are forwarded to the disk as a stream of 32k requests.
> >> > When the write is done from dom0 the disk gets a stream of 64k requests.
> >> > This would explain the slowdown.
> >> 
> >> Why isn't xbdback already aggregating these requests?
> >
> > I don't know; maybe because they don't come in order or something like
> > that ?
> 
> IIRC, the requests have to arrive in order and be picked up during the
> same invocation of the backend event handler.
> 
> So, if the frontend sends the first 32k and asserts the request event,
> if the backend can wake up and pull that request off the queue before
> the frontend writes the second 32k, they won't be merged.
> 
> This behavior was originally designed to deal with Linux, which under
> some poorly understood conditions would send a sequence of single-block
> IOs, and with the NetBSD Xen*2* domU code, which had a 64k MAXPHYS and
> would (again IIRC) break up the 64k into a 44k and a 20k, but send them
> as one update to the descriptor ring.

One reasonable adjustment might be to "nagle" requests which are exactly
44K, or exactly 32K (since it's the next power of 2 down): when receiving
such a request, yield once before clearing it, then check to see if the
next descriptor ring entry is contiguous.

-- 
Thor Lancelot Simon                                        
tls%rek.tjls.com@localhost
    "Even experienced UNIX users occasionally enter rm *.* at the UNIX
     prompt only to realize too late that they have removed the wrong
     segment of the directory structure." - Microsoft WSS whitepaper