Stephen Borrill wrote:
On Fri, 22 Oct 2010, Joerg Sonnenberger wrote:On Fri, Oct 22, 2010 at 11:01:03PM +0200, Manuel Bouyer wrote:Did you check if squid is hitting some ressource limit, maybe file descriptors ?
I'm not seeing any error in squid log that indicates these limits has been reached (something like "Warning your cache is running out of file descriptors" IIRC). We did increase this limit in squid but that doesn't help.
Or sockets in time wait state.
I've run netstat at multiple sites when the error occured but the number of local sockets in use seemed to vary greatly. If this is the case, what could we do?
Tuan (my co-worked) will correct me if I'm wrong, but it's proving to be an ipfilter problem. With ipfilter disabled, there are literally zero errors (we did up file descriptors to 8192 BTW).On a related issue, we are also seeing very slow rsync transfers under some circumstances. Again these are fixed by disabling ipfilter.The ipf.conf files are quite complex and are built up automatically from a machine-wide configuration file that includes things like:- ip address and netmasks - firewall security level - NAT on/offI've recommended to start from a simple ipf.conf (basically just pass in all/pass out all) and build up from there to see if we can work out what triggers the problem.
Oddly enough, as Stephen mentioned, disabling ipfilter seems to make the error go away completely. Could it be our firewall is dropping packets which makes squid think its parent proxy is dead? I'm working on a building up the firewall from a minimal set of rules to see if this is the case.
Digging into tech-net's archive from last year I found this post which looks like it might be related. Any thoughts on this?
http://mail-index.netbsd.org/tech-net/2009/11/17/msg001719.htmlBelow is a TCP conversation between squid with the upstream proxy during busy traffic. The part that's puzzling me is I'm seeing (repeated) identical RST packets (3 or 4 per TCP connection) sent by squid to the upstream proxy.
14:18:25.532933 IP netbsd.57601 > upstream.3129: S 3791930686:3791930686(0) win 32768 <mss 1460,nop,wscale 3,sackOK,nop,nop,nop,nop,timestamp 1 0> 14:18:25.538322 IP upstream.3129 > netbsd.57601: S 1766808328:1766808328(0) ack 3791930687 win 5792 <mss 1460,sackOK,timestamp 3382782028 1,nop,wscale 2> 14:18:25.538353 IP netbsd.57601 > upstream.3129: . ack 1 win 4197 <nop,nop,timestamp 1 3382782028> 14:18:25.538492 IP netbsd.57601 > upstream.3129: P 1:816(815) ack 1 win 4197 <nop,nop,timestamp 1 3382782028> 14:18:25.545471 IP upstream.3129 > netbsd.57601: . ack 816 win 1856 <nop,nop,timestamp 3382782034 1> 14:18:26.387247 IP netbsd.57601 > upstream.3129: F 816:816(0) ack 1 win 4197 <nop,nop,timestamp 3 3382782034> 14:18:26.434730 IP upstream.3129 > netbsd.57601: . ack 817 win 1856 <nop,nop,timestamp 3382782923 3> 14:18:28.606196 IP upstream.3129 > netbsd.57601: . 1:1449(1448) ack 817 win 1856 <nop,nop,timestamp 3382785095 3> 14:18:28.606197 IP upstream.3129 > netbsd.57601: P 1449:1553(104) ack 817 win 1856 <nop,nop,timestamp 3382785095 3> 14:18:28.606199 IP upstream.3129 > netbsd.57601: . 1553:3001(1448) ack 817 win 1856 <nop,nop,timestamp 3382785095 3> 14:18:28.606225 IP netbsd.57601 > upstream.3129: R 3791931503:3791931503(0) win 0 14:18:28.606238 IP netbsd.57601 > upstream.3129: R 3791931503:3791931503(0) win 0 14:18:28.606249 IP netbsd.57601 > upstream.3129: R 3791931503:3791931503(0) win 0
Thanks for all your help so far. Kind regards, Tuan