tech-net archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: ipfilter randomly dropping (ssh-)connections
On Tue, Jun 17, 2014 at 09:41:41PM +1000, Darren Reed wrote:
> On 12/06/2014 1:57 AM, Petar Bogdanovic wrote:
> > Hi,
> >
> > about a week ago, the automated daily ssh-tunnels to a netbsd-6 box
> > started to close shortly after they were established (client said
> > "Connection closed by remote host"; server said: "fatal: Write failed:
> > Network is unreachable"). A quick tcpdump revealed that the server side
> > at one point just FINs the connection and then spams the client with a
> > bunch of TCP resets.
> >
> > After a while of tcpdump and ktrace, disabling ipfilter (v4.1.34) solved
> > the problem. Which was very confusing, because its ipf.conf hasn't
> > changed for years.
> >
>
> Did anything change or has it really just started happening?
The software on that machine is basically an image generated from
multiple NetBSD filesets, pkgsrc packages and a git managed tree of
files and diffs. No manual changes are ever supposed to happen, just
indirect changes through regenerated images (which then replaces
everything except the data partition).
Therefore it's probably safe to say that "it just started happening".
Last regen was back in april due to the OpenSSL update. The problems
started appearing about two weeks ago. Of course it's possible that the
network environment has changed in the meantime (the system is hosted in
Germany) but neither I nor the support team there can see any problems
with the packet transport.
It doesn't always fail, though. When it fails, it fails after a very
short time.
Here is a list of the last n daily tunnels. As you can see the success
rate got better in the last few days (T=terminated, f=failed):
(...)
| 2014-06-02 16:00:06 | 66 | 3,459,222 | T |
| 2014-06-02 20:00:06 | 89 | 72,499,475 | T |
| 2014-06-03 00:00:06 | 97 | 3,562,668 | T |
| 2014-06-03 04:00:06 | 87 | 70,702,253 | T |
| 2014-06-03 08:00:07 | 53 | 1,482,232 | T |
| 2014-06-03 12:00:05 | 81 | 3,261,943 | T |
| 2014-06-03 16:00:06 | 110 | 72,548,463 | T |
| 2014-06-03 20:00:06 | 151 | 72,599,796 | T |
| 2014-06-04 00:00:06 | 117 | 4,214,858 | T |
| 2014-06-04 04:00:05 | 114 | 72,296,489 | T |
| 2014-06-04 08:00:06 | 71 | 2,970,687 | T |
| 2014-06-04 12:00:06 | 81 | 3,679,148 | T |
| 2014-06-04 16:00:06 | 79 | 3,824,554 | T |
| 2014-06-04 20:00:06 | 89 | 3,891,503 | T |
| 2014-06-05 00:00:06 | 112 | 6,565,550 | T |
| 2014-06-05 04:00:07 | 0 | 0 | f |
| 2014-06-05 08:00:06 | 116 | 71,642,789 | T |
| 2014-06-05 12:00:08 | 0 | 0 | f |
| 2014-06-05 16:00:05 | 0 | 0 | f |
| 2014-06-05 20:00:06 | 0 | 0 | f |
| 2014-06-06 00:00:06 | 0 | 0 | f |
| 2014-06-06 04:00:06 | 0 | 0 | f |
| 2014-06-06 08:00:05 | 2,261 | 86,091,470 | T |
| 2014-06-06 12:00:06 | 0 | 0 | f |
| 2014-06-06 16:00:06 | 5,009 | 3,666,629 | T |
| 2014-06-06 20:00:06 | 63 | 2,660,740 | T |
| 2014-06-07 00:00:05 | 0 | 0 | f |
| 2014-06-07 04:00:06 | 0 | 0 | f |
| 2014-06-07 08:00:07 | 0 | 0 | f |
| 2014-06-07 10:00:46 | 192 | 72,987,380 | T |
| 2014-06-07 12:00:07 | 0 | 0 | f |
| 2014-06-07 16:00:06 | 0 | 0 | f |
| 2014-06-07 20:00:06 | 0 | 0 | f |
| 2014-06-08 00:00:05 | 0 | 0 | f |
| 2014-06-08 04:00:06 | 0 | 0 | f |
| 2014-06-08 08:00:07 | 0 | 0 | f |
| 2014-06-08 12:00:05 | 0 | 0 | f |
| 2014-06-08 16:00:06 | 136 | 97,835,680 | T |
| 2014-06-08 20:00:06 | 50 | 71,133,165 | T |
| 2014-06-09 00:00:06 | 29 | 1,012,296 | T |
| 2014-06-09 04:00:06 | 0 | 0 | f |
| 2014-06-09 08:00:05 | 0 | 0 | f |
| 2014-06-09 12:00:07 | 44 | 1,314,205 | T |
| 2014-06-09 16:00:05 | 53 | 2,742,456 | T |
| 2014-06-09 20:00:06 | 74 | 2,533,244 | T |
| 2014-06-10 00:00:06 | 0 | 0 | f |
| 2014-06-10 04:00:06 | 0 | 0 | f |
| 2014-06-10 08:00:05 | 143 | 72,823,418 | T |
| 2014-06-10 12:00:06 | 0 | 0 | f |
| 2014-06-10 16:00:05 | 0 | 0 | f |
| 2014-06-10 20:00:06 | 0 | 0 | f |
| 2014-06-11 00:00:06 | 0 | 0 | f |
| 2014-06-11 04:00:05 | 0 | 0 | f |
| 2014-06-11 08:00:06 | 0 | 0 | f |
| 2014-06-11 12:00:05 | 219 | 73,231,862 | T |
| 2014-06-11 16:00:06 | 0 | 0 | f |
| 2014-06-11 20:00:05 | 0 | 0 | f |
| 2014-06-12 00:00:05 | 150 | 72,114,481 | T |
| 2014-06-12 04:00:05 | 98 | 71,960,509 | T |
| 2014-06-12 08:00:06 | 44 | 1,135,453 | T |
| 2014-06-12 12:00:06 | 78 | 3,051,787 | T |
| 2014-06-12 16:00:06 | 70 | 2,483,544 | T |
| 2014-06-12 20:00:06 | 79 | 3,227,264 | T |
| 2014-06-13 00:00:06 | 0 | 0 | f |
| 2014-06-13 12:00:09 | 5,216 | 102,545,019 | T |
| 2014-06-13 16:00:06 | 0 | 0 | f |
| 2014-06-13 20:00:06 | 0 | 0 | f |
| 2014-06-14 00:00:06 | 104 | 2,817,145 | T |
| 2014-06-14 04:00:07 | 0 | 0 | f |
| 2014-06-14 08:00:07 | 88 | 71,905,831 | T |
| 2014-06-14 12:00:05 | 54 | 2,427,955 | T |
| 2014-06-14 16:00:06 | 0 | 0 | f |
| 2014-06-14 20:00:05 | 56 | 2,967,200 | T |
| 2014-06-15 00:00:06 | 67 | 3,066,823 | T |
| 2014-06-15 04:00:06 | 0 | 0 | f |
| 2014-06-15 08:00:07 | 102 | 72,676,624 | T |
| 2014-06-15 12:00:06 | 42 | 1,728,665 | T |
| 2014-06-15 16:00:06 | 68 | 1,890,189 | T |
| 2014-06-15 20:00:06 | 88 | 72,158,931 | T |
| 2014-06-16 00:00:05 | 91 | 2,903,265 | T |
| 2014-06-16 04:00:06 | 73 | 71,246,715 | T |
| 2014-06-16 08:00:05 | 0 | 0 | f |
| 2014-06-16 16:00:06 | 157 | 72,627,958 | T |
| 2014-06-16 17:00:02 | 14 | 615,990 | T |
| 2014-06-16 17:00:43 | 13 | 614,651 | T |
| 2014-06-16 20:00:05 | 94 | 72,513,012 | T |
| 2014-06-17 00:00:05 | 68 | 71,048,464 | T |
| 2014-06-17 04:00:06 | 0 | 0 | f |
| 2014-06-17 08:00:05 | 107 | 74,812,559 | T |
| 2014-06-17 12:00:05 | 92 | 74,303,376 | T |
After seeing this and knowing that nothing has changed, I would be very
convinced that this issue has nothing to do with NetBSD and/or ipfilter
but whenever it happens, I can see that FIN leaving the interface.. :)
Home |
Main Index |
Thread Index |
Old Index