something is randomly closing ssh-tunnels (was: ipfilter randomly dropping..)

[Petar Bogdanovic <petar%smokva.net@localhost>]

During the past few weeks the ssh-tunnels to a remote machine started
failing randomly.  In a previous mail to tech-net I prematurely blamed
ipfilter because disabling it yielded some immediate success.

Unfortunately, subsequent testing showed that having npf enabled instead
eventually lead to the same issues.

What I know:

        * the server suddenly FINs the connection
        * the server ignores everything after that and sends about 20-30
          RSTs for lots of late ACKs sent by the client
        * ipmon is able to track the connection but misses the FIN
        * yet ipfilter manages to update its state table and reduces the
          TTL of the connection from 24h to 30s
        * a server-tcpdump captures the FIN
        * a client-tcpdump captures the same FIN
        * according to wireshark, the FINs in both pcaps have sequence
          numbers that indicate lost segments (which at least in one
          case makes little sense since it was captured directly at the
          source)
        * ssh and sshd both never try to tear down the connection
        * ssh reports that the remote end has closed the connection
        * sshd bails on a failed write() with ENETUNREACH
        * the success rate of the tunnel changes a lot, first it was 50%
          then it was 100% and during the past few days it was almost 0%
        * server and client are managed very carefully by me and there
          were no significant changes during the past 6+ months.

Now I'm thinking about compiling sshd with SO_DEBUG and a new kernel
with TCP_DEBUG but I'm not sure, what I'll be able to read from this.

Any suggestions are very much appreciated, since I'm running out of
ideas and would like to avoid messing with the (production-)server
more than is really necessary.

Thanks,

                Petar Bogdanovic


P.S. pcaps of one "failed" tunnel are here:
        http://smokva.net/pcap/crane.tgz