finally i built a xserver for my VAXStation 4000/90. But it causes
the kernel to dump core (well, on vax it actually doesn't...).
That probably indicates a kernel bug, I'd say; unless the server is
doing something dodgy like mucking about with /dev/mem, it shouldn't be
capable of panicking the kernel.
Your stack trace makes it look like a bug in the spx driver, at a
five-second glance. I can't say much more, because both your OS and
your X are versions I've never played with at all, much less on a VAX.
That aside....
I wouldn't think the Xorg Xserver would work on a VAX, so I'm not
surprised.
Perhaps not out of the box, but, unless they've broken something or the
framebuffer is unusually funky, it would be fairly easy to make it
work. While I've worked with MIT X, not x.org (and definitely not the
crawling design horror that is XFree86), I have done, for it, three
custom ddx layers - enough to know that getting a simple one working is
not really that complicated. (What were the three? NeXT 2bpp
greyscale, Sun S24, and Sun cg14.) Given hardware, leisure, and
motivation, I could probably do it within a day or two of having the
system set up and my tools installed, so I'd estimate a week or two for
someone decent but starting from scratch. (Unless the framebuffer
doesn't have a dumb-array-of-pixels mode, in which case it'd take me
longer.)