vncserver on DOM0, problem connecting from outside

[Joachim König <him%online.de@localhost>]

Hello,

I've installed Xen3 and a NetBSD dom0 as described by Joel Carnat in

http://www.tumfatig.net/docs/00_tut/virtual/NetBSD%20-%20virtual%20LAN%20with%20Xen3.php

which worked without problems, thanks, Joel.

I can now either boot NetBSD directly or via xen and a NetBSD dom0 (e.g. 
userland is identical).
I normally start a vncserver on NetBSD in order to connect with a 
tightvncclient from a windows box
via a ssh tunnel. This works with the direct boot without problems, but 
trying the same when running
a NetBSD dom0 results in an connection abort when vncclient tries to 
contact the server.

Actually it's sshd which aborts the connection. I started sshd in the 
foreground with debugging enabled
which shows:

> debug1: Setting controlling tty using TIOCSCTTY.
> debug1: server_input_channel_open: ctype direct-tcpip rchan 257 win 
> 16384 max 16384
> debug1: server_request_direct_tcpip: originator client-side-connection 
> port 0, target lo
> calhost port 5901
> debug1: getsockopt TCP_NODELAY: Connection reset by peer
> debug1: channel 3: new [direct-tcpip]
> debug1: server_input_channel_open: confirm direct-tcpip
> debug1: channel 3: not connected: Connection refused

that is, the connection is aborted when it tries to set the socket 
option TCP_NODELAY, here the
relevant bits from misc.c in crypto/dist/ssh:

> /* disable nagle on socket */
> void
> set_nodelay(int fd)
> {
>         int opt;
>         socklen_t optlen;
>
>         optlen = sizeof opt;
>         if (getsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &opt, &optlen) == 
> -1) {
>                 debug("getsockopt TCP_NODELAY: %.100s", strerror(errno));
>                 return;
>         }
>         if (opt == 1) {
>                 debug2("fd %d is TCP_NODELAY", fd);
>                 return;
>         }
>         opt = 1;
>         debug2("fd %d setting TCP_NODELAY", fd);
>         if (setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &opt, sizeof opt) 
> == -1)
>                 error("setsockopt TCP_NODELAY: %.100s", strerror(errno));
> }
I also commented out the getsockopt call to see if setsockopt woul work, 
but then the latter
fails in the same way. A telnet from the windows client to the ssh 
tunnel port produces the
same result, e.g. it's not the specific vnc client that causes the problem.

Does anybody have an idea why the getsockopt fails in a dom0 but works 
in a normal kernel?

Joachim