NetBSD+Xen with short-lived throw-away guests

[Torbjorn Granlund <tg%gmplib.org@localhost>]

I am developing a compiler teaching/judging system based on NetBSD+Xen
(as Dom0) where all execution of student compilers is done in Xen guests
(DomU).  For executing the code the student compilers generate from our
test suite, we use either Xen (for x86 and JVM targets) or qemu (for
ARM, SPARC, MIPS targets).

Before booting a Xen guest, a prepared master disk image is copied to a
throw-away image.  The throw-away image is used for one judging round,
then discarded.

The Xen master disk images are about 4 GB each.  They could be trimmed,
but not to a really small size.

My first attempt was to create a tmpfs for the throw-away image, then cp
from a master image to the tmpfs.  This copying is fast, more so since
the master image tend to live in buffer cache.

But to my surprise the NetBSD guest then runs poorly.  It creates 100%
load the Dom0 for several minutes while booting.  This DomU uses PV
virtualisation (in fact it uses the kernel made available for this
purpose by the NetBSD project, version 6.1.3).

Attempt 2 is more plain.  I just cp (or dd) the master image to a
throw-away image in the same filesystem.  The disk is an Intel S3500 SSD
(or actually two mirrored S3500).  This copy procedure takes about two
minutes, and also wears on the SSD (4 GB is written every 5 minutes
under full load).  But this naive solution is faster than the tmpfs
trick.

I haven't tried replacing tmpfs with mfs.  I haven't tried using HVM
instead of PV.

Do you have any advice on how to do this cleverly?

A copy-on-write on a per-file basis would be awesome...

The system is operational and used already, I just want to improve its
speed and decrease SSD wear.


Torbjörn