Re: something is randomly closing ssh-tunnels (was: ipfilter randomly dropping..)

[Petar Bogdanovic <petar%smokva.net@localhost>]

On Tue, Jun 24, 2014 at 10:44:12AM +0200, Daniel Hartmeier wrote:
> 
> Have you tried with all packet filters disabled, just to make sure?

When this started, I actually did disable ipfilter.  But back then the
failures were declining so when I finally decided to go filterless, the
problem was no longer reproducible---with or without ipf (or npf).

This week this changed and the tunnels were reliably failing again.. so
I did it again:

   | 2014-06-24 08:00:06 |     0 |              0 | f  |
   | 2014-06-24 09:12:27 |     0 |              0 | f  |
   | 2014-06-24 09:14:18 |     0 |              0 | f  |
   | 2014-06-24 09:22:51 |     0 |              0 | f  |
   | 2014-06-24 09:24:00 |     0 |              0 | f  |
   | 2014-06-24 09:46:48 |     0 |              0 | f  |
   | 2014-06-24 09:48:43 |     0 |              0 | f  |
   | 2014-06-24 09:50:16 |     0 |              0 | f  |
   | 2014-06-24 09:59:52 |     0 |              0 | f  |
   | 2014-06-24 10:05:08 |     0 |              0 | f  |
   | 2014-06-24 10:06:13 |     0 |              0 | f  |
   | 2014-06-24 10:14:20 |     0 |              0 | f  |
   | 2014-06-24 10:17:15 |     0 |              0 | f  |
   | 2014-06-24 10:22:56 |     0 |              0 | f  | <--- disabled ipf here
   | 2014-06-24 10:49:24 | 7,830 |  3,005,086,948 | T  | <--- successfuly 
transfered 3GB
   | 2014-06-24 10:22:56 |     0 |              0 | f  | <--- enabled ipf here
   | 2014-06-24 12:12:45 |     0 |              0 | f  |

So the local filter actually does play some role here.. :)


> On Mon, Jun 23, 2014 at 12:24:08PM +0200, Petar Bogdanovic wrote:
> 
> >     * sshd bails on a failed write() with ENETUNREACH
> 
> This can be because IPF (and NPF) return this errno to local sockets
> when blocking a packet.
> 
> Probably the blocked packet is part of the port forwarded connection,
> and sshd is reacting to the write error by closing the control
> connection.
> 
> How do you filter your loopback interface?
> 
> Do you have specific stateless pass rules for lo0?
> 
> I've seen cases where, without explicit lo0 rules, the packet filter
> tried to track state on loopback, and saw packets only in one direction,
> leading to stalling sessions iff when window scaling was enabled and
> traffic was just flowing fast enough to fill windows. This could work
> fine for weeks, just to pop up after something unrelated was changed
> (adding a database index to increase query response speed).

I've been using stateless pass rules for lo0 in both cases, ipf and npf.

My ipf configuration is older and might contain some ambiguities,
especially with regard to the lack of lo0 rules in ipf6.conf:

        # ipf.conf
        pass in  quick on lo0 all
        pass out quick on lo0 all

        block in quick all with mbcast

        block in  all
        block out all

        pass in  proto icmp icmp-type 3
        pass out proto icmp icmp-type 3

        pass in  proto icmp icmp-type 8 keep state
        pass out proto icmp icmp-type 8 keep state

        block return-rst  in proto tcp
        block return-icmp in proto udp

        (...)


        # ipf6.conf
        block in  quick all
        block out quick all


The npf configuration is simpler and makes no distinction between ip4
and ip6:

        # npf.conf
        $tcp_in  = { domain, ... }
        $udp_in  = { domain }

        $tcp_out = { domain, ... }
        $udp_out = { domain, ... }

        procedure "log" {
               # ifconfig npflog0 create
               log: npflog0
        }

        group (default) {
               pass final on lo0 all

               pass stateful in  final proto icmp icmp-type 8 all
               pass stateful out final proto icmp icmp-type 8 all

               pass stateful in  final proto tcp to any port $tcp_in
               pass stateful in  final proto udp to any port $udp_in
               pass stateful out final proto tcp to any port $tcp_out
               pass stateful out final proto udp to any port $udp_out

               block return all
               pass out proto icmp icmp-type 3 all
               pass out proto tcp flags R/R all
        }


Otherwise your interpretation is very intriguing given the fact that the
only connections failing are the ssh connections that do port-forwarding.

Worth mentioning: If transfers are short, they sometimes succeed even if
ipf is on.