tech-net archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

ipfilter randomly dropping (ssh-)connections



Hi,

about a week ago, the automated daily ssh-tunnels to a netbsd-6 box
started to close shortly after they were established (client said
"Connection closed by remote host"; server said: "fatal: Write failed:
Network is unreachable").  A quick tcpdump revealed that the server side
at one point just FINs the connection and then spams the client with a
bunch of TCP resets.

After a while of tcpdump and ktrace, disabling ipfilter (v4.1.34) solved
the problem.  Which was very confusing, because its ipf.conf hasn't
changed for years.

The following [1]issue seems similar.  I'm also attaching the [2]full
and [3]truncated pcaps of the failed ssh-session, and my [4]ipf.conf.

Maybe someone has some ideas about this.

Thanks,

                Petar Bogdanovic


[1]     http://sourceforge.net/p/ipfilter/bugs/5/

[2]     http://smokva.net/pcap/crane.tgz

[3]     tcpdump client (77.X.X.X):
        (...)
23:12:30.295355 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 
5578992:5580440, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.295358 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 
5580440:5581888, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.295360 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 
5581888:5583336, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.295361 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 
5583336:5584784, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.295363 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 
5584784:5586232, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.295365 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 
5586232:5587680, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.295374 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5581888, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.295378 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5584784, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.295382 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5587680, win 
11303, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.295393 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5587680, win 
12327, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.296126 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 
5587680:5589128, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.296128 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [.], seq 
5589128:5590576, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.296130 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [FP.], seq 
5592024:5592568, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 544
23:12:30.296139 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5590576, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.296145 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5590576, win 
12027, options [nop,nop,TS val 31 ecr 31,nop,nop,sack 1 {5592024:5592569}], 
length 0
23:12:30.296153 IP 77.X.X.X.65352 > 85.X.X.X.22: Flags [.], ack 5590576, win 
12389, options [nop,nop,TS val 31 ecr 31,nop,nop,sack 1 {5592024:5592569}], 
length 0
23:12:30.323294 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087322783, win 
0, length 0
23:12:30.329066 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087325679, win 
0, length 0
23:12:30.329067 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087328575, win 
0, length 0
23:12:30.329975 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087330023, win 
0, length 0
23:12:30.331049 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087332919, win 
0, length 0
23:12:30.333553 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087335815, win 
0, length 0
23:12:30.333554 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087338711, win 
0, length 0
23:12:30.333555 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087340159, win 
0, length 0
23:12:30.333556 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087343055, win 
0, length 0
23:12:30.333557 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087345951, win 
0, length 0
23:12:30.333558 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087348847, win 
0, length 0
23:12:30.333560 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087348847, win 
0, length 0
23:12:30.333561 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087351743, win 
0, length 0
23:12:30.333562 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087354639, win 
0, length 0
23:12:30.333563 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087356087, win 
0, length 0
23:12:30.338040 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087361879, win 
0, length 0
23:12:30.338041 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087364775, win 
0, length 0
23:12:30.338042 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087364775, win 
0, length 0
23:12:30.338043 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087367671, win 
0, length 0
23:12:30.338044 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087367671, win 
0, length 0
23:12:30.338046 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087367671, win 
0, length 0
23:12:30.338047 IP 85.X.X.X.22 > 77.X.X.X.65352: Flags [R], seq 3087358983, win 
0, length 0

        tcpdump server (85.X.X.X):
        (...)
23:12:30.259749 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5492112, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.265152 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5495008, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.265157 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5492112, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.265405 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 
5578992:5580440, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.265418 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 
5580440:5581888, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.265433 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 
5581888:5583336, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.265449 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 
5583336:5584784, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.265460 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 
5584784:5586232, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.265475 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 
5586232:5587680, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.265490 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 
5587680:5589128, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.265505 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [.], seq 
5589128:5590576, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 1448
23:12:30.265806 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5499352, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.265831 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5497904, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.266249 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [FP.], seq 
5592024:5592568, ack 6391120, win 10341, options [nop,nop,TS val 31 ecr 31], 
length 544
23:12:30.269507 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5502248, win 
12208, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.269516 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5505144, win 
11846, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.270483 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5505144, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.275423 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5508040, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.276292 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5510936, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.277578 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5510936, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.285405 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5513832, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.285804 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5516728, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.285809 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5518176, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.286450 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5523968, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.286463 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5521072, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.286468 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5525416, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.286476 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5528312, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.286494 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5529760, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.292326 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5532656, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.292343 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5535552, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293450 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5535552, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293474 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5538448, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293788 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5541344, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293815 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5542792, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293832 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5545688, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293838 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5545688, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.293865 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [P.], seq 
6391120:6391168, ack 5545688, win 12389, options [nop,nop,TS val 31 ecr 31], 
length 48
23:12:30.293902 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087322783, win 
0, length 0
23:12:30.299449 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5548584, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.299461 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5551480, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.299473 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087325679, win 
0, length 0
23:12:30.299483 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087328575, win 
0, length 0
23:12:30.300303 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5552928, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.300314 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087330023, win 
0, length 0
23:12:30.301484 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5555824, win 
12208, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.301497 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087332919, win 
0, length 0
23:12:30.303800 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5558720, win 
11846, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303813 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087335815, win 
0, length 0
23:12:30.303858 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5561616, win 
11484, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303864 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5563064, win 
12327, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303871 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5565960, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303877 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5568856, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303892 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087338711, win 
0, length 0
23:12:30.303903 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5571752, win 
11303, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303909 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5571752, win 
12327, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303917 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5574648, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303932 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087340159, win 
0, length 0
23:12:30.303944 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5577544, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303949 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5578992, win 
12389, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.303968 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087343055, win 
0, length 0
23:12:30.303978 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087345951, win 
0, length 0
23:12:30.303988 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087348847, win 
0, length 0
23:12:30.303999 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087348847, win 
0, length 0
23:12:30.304009 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087351743, win 
0, length 0
23:12:30.304019 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087354639, win 
0, length 0
23:12:30.304029 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087356087, win 
0, length 0
23:12:30.308639 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5584784, win 
11665, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.308646 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5587680, win 
11303, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.308658 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087361879, win 
0, length 0
23:12:30.308668 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087364775, win 
0, length 0
23:12:30.308684 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5587680, win 
12327, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.308700 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5590576, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.308708 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087364775, win 
0, length 0
23:12:30.308718 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087367671, win 
0, length 0
23:12:30.308735 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5590576, win 
12027, options [nop,nop,TS val 31 ecr 31,nop,nop,sack 1 {5592024:5592569}], 
length 0
23:12:30.308753 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087367671, win 
0, length 0
23:12:30.308767 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5590576, win 
12389, options [nop,nop,TS val 31 ecr 31,nop,nop,sack 1 {5592024:5592569}], 
length 0
23:12:30.308780 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087367671, win 
0, length 0
23:12:30.308791 IP 77.X.X.X.50772 > 85.X.X.X.22: Flags [.], ack 5581888, win 
12027, options [nop,nop,TS val 31 ecr 31], length 0
23:12:30.308803 IP 85.X.X.X.22 > 77.X.X.X.50772: Flags [R], seq 3087358983, win 
0, length 0

[4]     ipf.conf
###
pass in  quick on lo0 all
pass out quick on lo0 all

###
block in quick all with mbcast

###
block in quick from 192.168.0.0/16 to any
block in quick from 172.16.0.0/12 to any
block in quick from 10.0.0.0/8 to any
block in quick from 127.0.0.0/8 to any
block in quick from 0.0.0.0/8 to any
block in quick from 169.254.0.0/16 to any
block in quick from 192.0.2.0/24 to any
block in quick from 204.152.64.0/23 to any
block in quick from 224.0.0.0/3 to any

###
block out quick from any to 192.168.0.0/16
block out quick from any to 172.16.0.0/12
block out quick from any to 10.0.0.0/8
block out quick from any to 127.0.0.0/8
block out quick from any to 0.0.0.0/8
block out quick from any to 169.254.0.0/16
block out quick from any to 192.0.2.0/24
block out quick from any to 204.152.64.0/23
block out quick from any to 224.0.0.0/3

###
block in  all
block out all

###
pass in  proto icmp icmp-type 3
pass out proto icmp icmp-type 3

###
pass in  proto icmp icmp-type 8 keep state
pass out proto icmp icmp-type 8 keep state

###
block return-rst  in proto tcp
block return-icmp in proto udp

###
pass in  proto esp from any to any
pass out proto esp from any to any

###
pass in  proto tcp from any to any port = 22000 flags S keep state keep frags
pass in  proto tcp from any to any port = ssh flags S keep state keep frags

###
pass in  proto udp from any to any port = domain keep state
pass out proto udp from any to any port = domain keep state
pass in  proto tcp from any to any port = domain flags S keep state keep frags
pass out proto tcp from any to any port = domain flags S keep state keep frags

###
pass in  proto tcp from any to any port = http flags S keep state keep frags

###
pass in  proto tcp from any to any port = smtp flags S keep state keep frags
pass out proto tcp from any to any port = smtp flags S keep state keep frags

###
pass in  proto tcp from any to any port = submission flags S keep state keep 
frags

###
pass in  proto tcp from any to any port = imap flags S keep state keep frags
pass in  proto tcp from any to any port = imaps flags S keep state keep frags

###
pass out proto udp from any to any port = ntp keep state
pass out proto tcp from any to any port = ntp flags S keep state keep frags

###
pass out proto udp from any to any port = 6277 keep state

###
pass out proto tcp from any to any port = 2703 flags S keep state keep frags


Home | Main Index | Thread Index | Old Index