Re: troubleshooting I/O performance bottleneck

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Fri, May 01, 2009 at 11:58:52AM -0400, Thor Lancelot Simon wrote:
> On Thu, Apr 30, 2009 at 07:16:07PM -0400, Jed Davis wrote:
> > Manuel Bouyer <bouyer%antioche.eu.org@localhost> writes:
> > 
> > > On Tue, Apr 28, 2009 at 08:13:36AM -0400, Thor Lancelot Simon wrote:
> > >> On Tue, Apr 28, 2009 at 01:26:01PM +0200, Manuel Bouyer wrote:
> > >> > 
> > >> > This is also what I get on my test box. iostat -x shows that the vnd0
> > >> > device is getting write requests of 32k (this is expected as the
> > >> > xbd interface can't do 64k transfers so MAXPHYS is set to 32k for 
> > >> > domUs),
> > >> > and writes are forwarded to the disk as a stream of 32k requests.
> > >> > When the write is done from dom0 the disk gets a stream of 64k 
> > >> > requests.
> > >> > This would explain the slowdown.
> > >> 
> > >> Why isn't xbdback already aggregating these requests?
> > >
> > > I don't know; maybe because they don't come in order or something like
> > > that ?
> > 
> > IIRC, the requests have to arrive in order and be picked up during the
> > same invocation of the backend event handler.
> > 
> > So, if the frontend sends the first 32k and asserts the request event,
> > if the backend can wake up and pull that request off the queue before
> > the frontend writes the second 32k, they won't be merged.
> > 
> > This behavior was originally designed to deal with Linux, which under
> > some poorly understood conditions would send a sequence of single-block
> > IOs, and with the NetBSD Xen*2* domU code, which had a 64k MAXPHYS and
> > would (again IIRC) break up the 64k into a 44k and a 20k, but send them
> > as one update to the descriptor ring.
> 
> One reasonable adjustment might be to "nagle" requests which are exactly
> 44K, or exactly 32K (since it's the next power of 2 down): when receiving
> such a request, yield once before clearing it, then check to see if the
> next descriptor ring entry is contiguous.

Better would be to fix the domU to use 64k MAXPHYS again, and have the
xbd driver split the request if needed.

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--