Re: Frustrating network latency issue on NetBSD 6.1

[Mayuresh <mayuresh%acm.org@localhost>]

On Thu, Jun 20, 2013 at 09:38:34PM -0400, Greg Troxel wrote:
> The next thing I'd want to know is if the loss is in transmitted or
> received packets.  To do that you'll need to be able to tcpdump on the
> target, or get at counters. (Look at netstat -s; it's often useful to
> capture netstat -s, do something, re-capture, and diff).

Attaching ns1 and ns2. Two snapshots in between about 15 pings happened.

Will use tcpdump if above is not giving sufficient inputs.

Thanks for the help.

Mayuresh.

icmp:
        0 calls to icmp_error
        0 errors not generated because old message was icmp
        Output histogram:
                photuris: 86
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                echoreply: 86
                photuris: 3298669537024
        0 message responses generated
        0 path MTU changes
igmp:
        0 messages received
        0 messages received with too few bytes
        0 messages received with bad checksum
        0 membership queries received
        0 membership queries received with invalid field(s)
        0 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        0 membership reports sent
tcp:
        4082 packets sent
                1883 data packets (510427 bytes)
                4 data packets (1499 bytes) retransmitted
                2012 ack-only packets (2345 delayed)
                0 URG only packets
                0 window probe packets
                49 window update packets
                136 control packets
                0 send attempts resulted in self-quench
        4670 packets received
                1633 acks (for 506845 bytes)
                50 duplicate acks
                0 acks for unsent data
                3548 packets (2715270 bytes) received in-sequence
                3 completely duplicate packets (2860 bytes)
                0 old duplicate packets
                11 packets with some dup. data (3043 bytes duped)
                139 out-of-order packets (72642 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                25 window update packets
                20 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
        55 connection requests
        24 connection accepts
        78 connections established (including accepts)
        91 connections closed (including 13 drops)
        1 embryonic connection dropped
        0 delayed frees of tcpcb
        1686 segments updated rtt (of 1557 attempts)
        21 retransmit timeouts
                0 connections dropped by rexmit timeout
        0 persist timeouts (resulting in 0 dropped connections)
        1 keepalive timeout
                0 keepalive probes sent
                1 connection dropped by keepalive
        277 correct ACK header predictions
        2547 correct data packet header predictions
        528 PCB hash misses
        19 dropped due to no socket
        0 connections drained due to memory shortage
        2 PMTUD blackholes detected
        0 bad connection attempts
        24 SYN cache entries added
                0 hash collisions
                24 completed
                0 aborted (no space to build PCB)
                0 timed out
                0 dropped due to overflow
                0 dropped due to bucket overflow
                0 dropped due to RST
                0 dropped due to ICMP unreachable
                24 delayed free of SYN cache entries
        0 SYN,ACKs retransmitted
        0 duplicate SYNs received for entries already in the cache
        0 SYNs dropped (no route or no space)
        0 packets with bad signature
        0 packets with good signature
        0 successful ECN handshakes
        0 packets with ECN CE bit
        0 packets ECN ECT(0) bit
udp:
        60 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        0 dropped due to no socket
        0 broadcast/multicast datagrams dropped due to no socket
        0 dropped due to full socket buffers
        60 delivered
        38 PCB hash misses
        115 datagrams output
ip:
        3129 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped (out of ipqent)
        0 malformed fragments dropped
        0 fragments dropped after timeout
        0 packets reassembled ok
        3129 packets for this host
        0 packets for unknown/unsupported protocol
        0 packets forwarded (0 packets fast forwarded)
        0 packets not forwardable
        0 redirects sent
        0 packets no matching gif found
        2592 packets sent from this host
        44 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 datagrams with bad address in header
ip6:
        1687 total packets received
        0 with size smaller than minimum
        0 with data size < data length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 fragments that exceeded limit
        0 packets reassembled ok
        1687 packets for this host
        0 packets forwarded
        0 packets fast forwarded
        0 fast forward flows
        0 packets not forwardable
        0 redirects sent
        1699 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        5 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 packets that violated scope rules
        0 multicast packets which we don't join
        Input packet histogram:
                TCP: 1687
        Mbuf statistics:
                1281 one mbufs
                two or more mbuf:
                        lo0 = 406
                0 one ext mbufs
                0 two or more ext mbufs
        0 packets whose headers are not continuous
        0 tunneling packets that can't find gif
        0 packets discarded due to too many headers
        0 failures of source address selection
        1685 forward cache hit
        2 forward cache miss
icmp6:
        0 calls to icmp6_error
        0 errors not generated because old message was icmp6 or so
        0 errors not generated because of rate limitation
        Output packet histogram:
                multicast listener report: 10
                neighbor solicitation: 2
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        Histogram of error messages to be generated:
                0 no route
                0 administratively prohibited
                0 beyond scope
                0 address unreachable
                0 port unreachable
                0 packet too big
                0 time exceed transit
                0 time exceed reassembly
                0 erroneous header field
                0 unrecognized next header
                0 unrecognized option
                0 redirect
                0 unknown
        0 message responses generated
        0 messages with too many ND options
        0 messages with bad ND options
        0 bad neighbor solicitation messages
        0 bad neighbor advertisement messages
        0 bad router solicitation messages
        0 bad router advertisement messages
        0 router advertisement routes dropped
        0 bad redirect messages
        0 path MTU changes
tcp6:
        4082 packets sent
                1883 data packets (510427 bytes)
                4 data packets (1499 bytes) retransmitted
                2012 ack-only packets (2345 delayed)
                0 URG only packets
                0 window probe packets
                49 window update packets
                136 control packets
                0 send attempts resulted in self-quench
        4670 packets received
                1633 acks (for 506845 bytes)
                50 duplicate acks
                0 acks for unsent data
                3548 packets (2715270 bytes) received in-sequence
                3 completely duplicate packets (2860 bytes)
                0 old duplicate packets
                11 packets with some dup. data (3043 bytes duped)
                139 out-of-order packets (72642 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                25 window update packets
                20 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
        55 connection requests
        24 connection accepts
        78 connections established (including accepts)
        91 connections closed (including 13 drops)
        1 embryonic connection dropped
        0 delayed frees of tcpcb
        1686 segments updated rtt (of 1557 attempts)
        21 retransmit timeouts
                0 connections dropped by rexmit timeout
        0 persist timeouts (resulting in 0 dropped connections)
        1 keepalive timeout
                0 keepalive probes sent
                1 connection dropped by keepalive
        277 correct ACK header predictions
        2547 correct data packet header predictions
        528 PCB hash misses
        19 dropped due to no socket
        0 connections drained due to memory shortage
        2 PMTUD blackholes detected
        0 bad connection attempts
        24 SYN cache entries added
                0 hash collisions
                24 completed
                0 aborted (no space to build PCB)
                0 timed out
                0 dropped due to overflow
                0 dropped due to bucket overflow
                0 dropped due to RST
                0 dropped due to ICMP unreachable
                24 delayed free of SYN cache entries
        0 SYN,ACKs retransmitted
        0 duplicate SYNs received for entries already in the cache
        0 SYNs dropped (no route or no space)
        0 packets with bad signature
        0 packets with good signature
        0 successful ECN handshakes
        0 packets with ECN CE bit
        0 packets ECN ECT(0) bit
udp6:
        0 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        0 with no checksum
        0 dropped due to no socket
        0 multicast datagrams dropped due to no socket
        0 dropped due to full socket buffers
        0 delivered
        0 datagrams output
pim6:
        0 messages received
        0 messages received with too few bytes
        0 messages received with bad checksum
        0 messages received with bad version
        0 registers received
        0 bad registers received
        0 registers sent
rip6:
        0 messages received
        0 checksum calculations on inbound
        0 messages with bad checksum
        0 messages dropped due to no socket
        0 multicast messages dropped due to no socket
        0 messages dropped due to full socket buffers
        0 delivered
        0 datagrams output
arp:
        30 packets sent
                14 reply packets
                16 request packets
        18 packets received
                1 reply packet
                17 valid request packets
                3 broadcast/multicast packets
                0 packets with unknown protocol type
                0 packets with bad (short) length
                0 packets with null target IP address
                0 packets with null source IP address
                0 could not be mapped to an interface
                0 packets sourced from a local hardware address
                0 packets with a broadcast source hardware address
                0 duplicates for a local IP address
                0 attempts to overwrite a static entry
                0 packets received on wrong interface
                0 entrys overwritten
                0 changes in hardware address length
        14 packets deferred pending ARP resolution
                1 sent
                12 dropped
        0 failures to allocate llinfo
ddp:
        0 packets with short headers 
        0 packets with long headers 
        0 packets with no checksum 
        0 packets too short 
        0 packets with bad checksum 
        0 packets with not enough data 
        0 packets forwarded 
        0 packets encapsulated 
        0 packets rcvd for unreachable dest 
        0 packets dropped due to no socket space 
TP not configured

icmp:
        0 calls to icmp_error
        0 errors not generated because old message was icmp
        Output histogram:
                photuris: 118
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                echoreply: 118
                photuris: 3298669537024
        0 message responses generated
        0 path MTU changes
igmp:
        0 messages received
        0 messages received with too few bytes
        0 messages received with bad checksum
        0 membership queries received
        0 membership queries received with invalid field(s)
        0 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        0 membership reports sent
tcp:
        4083 packets sent
                1883 data packets (510427 bytes)
                4 data packets (1499 bytes) retransmitted
                2012 ack-only packets (2345 delayed)
                0 URG only packets
                0 window probe packets
                49 window update packets
                137 control packets
                0 send attempts resulted in self-quench
        4670 packets received
                1633 acks (for 506845 bytes)
                50 duplicate acks
                0 acks for unsent data
                3548 packets (2715270 bytes) received in-sequence
                3 completely duplicate packets (2860 bytes)
                0 old duplicate packets
                11 packets with some dup. data (3043 bytes duped)
                139 out-of-order packets (72642 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                25 window update packets
                20 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
        55 connection requests
        24 connection accepts
        78 connections established (including accepts)
        91 connections closed (including 13 drops)
        1 embryonic connection dropped
        0 delayed frees of tcpcb
        1686 segments updated rtt (of 1557 attempts)
        22 retransmit timeouts
                0 connections dropped by rexmit timeout
        0 persist timeouts (resulting in 0 dropped connections)
        1 keepalive timeout
                0 keepalive probes sent
                1 connection dropped by keepalive
        277 correct ACK header predictions
        2547 correct data packet header predictions
        528 PCB hash misses
        19 dropped due to no socket
        0 connections drained due to memory shortage
        2 PMTUD blackholes detected
        0 bad connection attempts
        24 SYN cache entries added
                0 hash collisions
                24 completed
                0 aborted (no space to build PCB)
                0 timed out
                0 dropped due to overflow
                0 dropped due to bucket overflow
                0 dropped due to RST
                0 dropped due to ICMP unreachable
                24 delayed free of SYN cache entries
        0 SYN,ACKs retransmitted
        0 duplicate SYNs received for entries already in the cache
        0 SYNs dropped (no route or no space)
        0 packets with bad signature
        0 packets with good signature
        0 successful ECN handshakes
        0 packets with ECN CE bit
        0 packets ECN ECT(0) bit
udp:
        60 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        0 dropped due to no socket
        0 broadcast/multicast datagrams dropped due to no socket
        0 dropped due to full socket buffers
        60 delivered
        38 PCB hash misses
        115 datagrams output
ip:
        3161 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped (out of ipqent)
        0 malformed fragments dropped
        0 fragments dropped after timeout
        0 packets reassembled ok
        3161 packets for this host
        0 packets for unknown/unsupported protocol
        0 packets forwarded (0 packets fast forwarded)
        0 packets not forwardable
        0 redirects sent
        0 packets no matching gif found
        2610 packets sent from this host
        61 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 datagrams with bad address in header
ip6:
        1687 total packets received
        0 with size smaller than minimum
        0 with data size < data length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 fragments that exceeded limit
        0 packets reassembled ok
        1687 packets for this host
        0 packets forwarded
        0 packets fast forwarded
        0 fast forward flows
        0 packets not forwardable
        0 redirects sent
        1699 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        5 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 packets that violated scope rules
        0 multicast packets which we don't join
        Input packet histogram:
                TCP: 1687
        Mbuf statistics:
                1281 one mbufs
                two or more mbuf:
                        lo0 = 406
                0 one ext mbufs
                0 two or more ext mbufs
        0 packets whose headers are not continuous
        0 tunneling packets that can't find gif
        0 packets discarded due to too many headers
        0 failures of source address selection
        1685 forward cache hit
        2 forward cache miss
icmp6:
        0 calls to icmp6_error
        0 errors not generated because old message was icmp6 or so
        0 errors not generated because of rate limitation
        Output packet histogram:
                multicast listener report: 10
                neighbor solicitation: 2
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        Histogram of error messages to be generated:
                0 no route
                0 administratively prohibited
                0 beyond scope
                0 address unreachable
                0 port unreachable
                0 packet too big
                0 time exceed transit
                0 time exceed reassembly
                0 erroneous header field
                0 unrecognized next header
                0 unrecognized option
                0 redirect
                0 unknown
        0 message responses generated
        0 messages with too many ND options
        0 messages with bad ND options
        0 bad neighbor solicitation messages
        0 bad neighbor advertisement messages
        0 bad router solicitation messages
        0 bad router advertisement messages
        0 router advertisement routes dropped
        0 bad redirect messages
        0 path MTU changes
tcp6:
        4083 packets sent
                1883 data packets (510427 bytes)
                4 data packets (1499 bytes) retransmitted
                2012 ack-only packets (2345 delayed)
                0 URG only packets
                0 window probe packets
                49 window update packets
                137 control packets
                0 send attempts resulted in self-quench
        4670 packets received
                1633 acks (for 506845 bytes)
                50 duplicate acks
                0 acks for unsent data
                3548 packets (2715270 bytes) received in-sequence
                3 completely duplicate packets (2860 bytes)
                0 old duplicate packets
                11 packets with some dup. data (3043 bytes duped)
                139 out-of-order packets (72642 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                25 window update packets
                20 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
        55 connection requests
        24 connection accepts
        78 connections established (including accepts)
        91 connections closed (including 13 drops)
        1 embryonic connection dropped
        0 delayed frees of tcpcb
        1686 segments updated rtt (of 1557 attempts)
        22 retransmit timeouts
                0 connections dropped by rexmit timeout
        0 persist timeouts (resulting in 0 dropped connections)
        1 keepalive timeout
                0 keepalive probes sent
                1 connection dropped by keepalive
        277 correct ACK header predictions
        2547 correct data packet header predictions
        528 PCB hash misses
        19 dropped due to no socket
        0 connections drained due to memory shortage
        2 PMTUD blackholes detected
        0 bad connection attempts
        24 SYN cache entries added
                0 hash collisions
                24 completed
                0 aborted (no space to build PCB)
                0 timed out
                0 dropped due to overflow
                0 dropped due to bucket overflow
                0 dropped due to RST
                0 dropped due to ICMP unreachable
                24 delayed free of SYN cache entries
        0 SYN,ACKs retransmitted
        0 duplicate SYNs received for entries already in the cache
        0 SYNs dropped (no route or no space)
        0 packets with bad signature
        0 packets with good signature
        0 successful ECN handshakes
        0 packets with ECN CE bit
        0 packets ECN ECT(0) bit
udp6:
        0 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        0 with no checksum
        0 dropped due to no socket
        0 multicast datagrams dropped due to no socket
        0 dropped due to full socket buffers
        0 delivered
        0 datagrams output
pim6:
        0 messages received
        0 messages received with too few bytes
        0 messages received with bad checksum
        0 messages received with bad version
        0 registers received
        0 bad registers received
        0 registers sent
rip6:
        0 messages received
        0 checksum calculations on inbound
        0 messages with bad checksum
        0 messages dropped due to no socket
        0 multicast messages dropped due to no socket
        0 messages dropped due to full socket buffers
        0 delivered
        0 datagrams output
arp:
        30 packets sent
                14 reply packets
                16 request packets
        18 packets received
                1 reply packet
                17 valid request packets
                3 broadcast/multicast packets
                0 packets with unknown protocol type
                0 packets with bad (short) length
                0 packets with null target IP address
                0 packets with null source IP address
                0 could not be mapped to an interface
                0 packets sourced from a local hardware address
                0 packets with a broadcast source hardware address
                0 duplicates for a local IP address
                0 attempts to overwrite a static entry
                0 packets received on wrong interface
                0 entrys overwritten
                0 changes in hardware address length
        14 packets deferred pending ARP resolution
                1 sent
                12 dropped
        0 failures to allocate llinfo
ddp:
        0 packets with short headers 
        0 packets with long headers 
        0 packets with no checksum 
        0 packets too short 
        0 packets with bad checksum 
        0 packets with not enough data 
        0 packets forwarded 
        0 packets encapsulated 
        0 packets rcvd for unreachable dest 
        0 packets dropped due to no socket space 
TP not configured