Re: nd6_free assumes all routers are processed by kernel RA

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Fri, 23 Aug 2019 20:43:53 +0100
    From:        Roy Marples <roy%marples.name@localhost>
    Message-ID:  <31fc6489-907d-dbef-af0e-d90d47aada61%marples.name@localhost>

  | We should do something with ICMP unreachable maybe? I assume the router 
  | did send something along those lines back?

It turns out to not have taken very long at all ... when I next went to
look to see if the default route was back, it was, so I tcpdump'd the
interface, and started a ssh to mail.netbsd.org

The short answer is yes, there is an ICMP unreachable (after each SYN
attempt).

But before the rest of that, replies to your subsequent reply...

roy%marples.name@localhost said:
  | I think you're over compliating it.

That's always possible.

  | We have an address on a prefix.

No, we don't, not really - we have an LL address, which has no prefix,
(in the usual sense) just a link that owns it.

  | We want to send to an address not on the prefix.

We want to send to an address, we have no idea where that is (as
in the kernel does not).   It might be on the local link, it might
be anywhere.   It certainly has a prefix that is not LL though.

  | We have a default route.

Yes.

  | Given these conditions I don't see any alternative but to send the  packet.

To attempt to send the packet, yes - to use the default route, no.

All we have is the LL address to use as the source addr.   The rules
for v6 say that routers *must not* forward packets using LL addresses.
Hence, sending such a packet to a router is guaranteed to fail (unless
the router is perverse, and doing v6 NAT perhaps - and I assure you,
nothing I have will ever do that.)

Sending the packet by first sending a NS for the target addr, to see if it
happens to be on the local link (about which we have no idea what prefix
may have been allocated) would be a reasonable thing to do - if we get
a reply (including from the router, acting as a proxy-NDP-server if that
is what it wants to do) then we can go ahead and send the packet.   But
sending it to a router, and just hoping that the router breaks the v6
forwarding rules seems wrong.

  | Otherwise it's not a default route - it's a route that might or might not
  | send your packet.

Well, that's always possible, regardless.... It is a default route but
only for routable packets.   Packets containing LL addresses are not.

[On when the default v6 route returns:]

roy%marples.name@localhost said:
  | It *should* be each RA that advertises itself as a default router.

That I think it is not.   As you will see below, RAs appear to be
being sent approx every 4 seconds (another parameter not able to be
varied, as best I can see ... as I said, cheap router) and the default
route certainly doesn't come back that quickly, if it did, I'd never be
able to do almost anything within reasonable time limits.

When I manually delete the route, it stays gone for (at least) hours,
sometimes days.   After the experiment for which the packet trace is below,
I deleted the default route (almost immediately) and it is still gone,
and while that's only a bit more than 1 hour so far, that's about 900
RA packets that have probably been received in the interim.   I have
no reason to believe that any one RA is different from any other, though
I suppose one that results from an explicit RS could be different (no RS's
were sent during the few minutes of my experiment - aside from running
dhcpcd with -n (which I certainly do not often do, if ever) or restarting
it (which I also don't often do, though I have once or twice when it
seemed slow detecting a new SSID connection and restarting (v4) DHCP to
get a new addr ... but not while I am just at home), is there anything
else which would cause dhcpcd to send a new RS (given RA's keep arriving
sent to the all hosts multicast addr) of its own volition?

While iwm is not the most stable of wireless interfaces around (our driver
seems a bit lacking) and from time to time needs much love and attention,
there have been no problems beween when I last deleted the v6 default route,
and where I did a "route get -inet6 default" to observe it had returned, in
fact, in my window where I have a root sh, there were no intervening
commands at all, and the interface suffered no interruptions (nothing more
serious that my ISP randomly changing the v4 addr it uses - which is not
the one my router gets told I believe, I think there are 2 levels of NAT,
one in my router, from my local net 10, to the addr the ISP issues it, and
another somewhere in the ISP net before it lets packets into the great
unknown .. the latter is the one which changes, killing all existing v4
connections, far too often!   None of that is relevant here though.

roy%marples.name@localhost said:
  | If you use dhcpcd, then `dhcpcd -n` should be enough to force a
  | soliciation and get the default route back.

That is not generally something I want to accomplish, and as above,
the default was back all by itself when I looked, so I did not
need to try it this time.

  | If not, try restarting dhcpcd and file a bug with me.

OK, though I have no reason to suspect that it would not work ... if
for some reason I need to force the default to be added back (by dhcpcd)
I'll try the -n version, and if it doesn't work, do that and let you
know.   My hope would be that the circumstances requiring that will
never occur...

roy%marples.name@localhost said:
  | OR just ping6 www.google.com. 

I could have done that, and that would have shown the ICMPv6 exists I
presume, but would have lessened the overall effect of demonstrating the
annoyance caused by the current behaviour...  (that is, I wanted to also
show the eventual fallback to v4).   Of course, I could have used any
other site than netbsd.org (the connection doesn't actually need to
succeed, just the TCP SYN to get through, even a RST reply would be fine)
and I could have used a dest port other than 22 ... this just happens to
be what I do all the time (I get the same kind of effect from my browser
aimed at www.netbsd.org .. though I suspect it might remember what addr
worked last time, and use it again, for a while at least, as once one
web page gets fetched, others tend to not suffer the delays - if the default
v6 route happens to exist, if it doesn't v4 gets used very quickly, and
all is OK.)


roy%marples.name@localhost said:
  | I don't think NetBSD.org IPv6 is entirely setup correctly - it's like a
  | blackhole when I ssh.

I have no issues with it at all from hosts (including my laptop when
applicable) which do have proper v6 addresses.   I use v6 to netbsd hosts
all the time.   The copy of this mail to the list will most likely use v6
from my MTA (which has real v6) to the netbsd mail server.

But even if there was some issue (I have seen no signs of anything wrong)
it would be irrelevant here - the nature of the issue I have is that no
v6 packet can possibly leave my local net (unless I was to set up a tunnel,
and I have no actual requirement to justify that right now) - so the state
of v6 config at netbsd.org (or anywhere else) is completely irrelevant - no
v6 packets from me can ever reach them (and no v6, or for that matter, any
other, address for my laptop is advertised in the DNS, so nothing is going
to be attempting to make incoming connections ... unless I take pains to
find out, I have no idea what my (real) v4 addr is - except that I know
that by tomorrow, the one in use today will no longer be it).

All of the issues are completely local - what the router does (of which only
sending the RA at all is an issue, everything else it is doing seems correct,
if a little over enthusiastic with its RA packets) and what my NetBSD laptop
does.   The only one of those I can alter (short term) is what my laptop does.


So, back to the packet dump:

The relevant part (I hope I didn't delete anything important, I have the
binary packet dump and can dump it again as often as needed, with -v
added if that would be of any use ... but I suspect not) of the packet
trace is below.

It starts with a couple of RA packets (I have left all of those included
here, if for no other purpose than to show time passing...) then the DNS
lookups (AAAA and A) for mail.netbsd.org, then a SYN attempt, ICMPv6 reply
(after a NDP sequence so the router can find my laptop's MAC address ...
you'd think that would not really be needed here, but never mind), followed
by a delay, a new v6 SYN attempt, immediate ICMPv6, delay, 3rd v6 SYN
attempt, ICMPv6, more delay (including a refresh of the router's NDP cache
for my laptop) and eventually a v4 SYN attempt (which succeeds) 75 seconds
(approx) after the sequence started.   I left in the 3-way v4 TCP handshake,
but didn't keep any packets after that (nothing interesting there).  Other
irrelevant traffic passing through the interface has also been deleted from
the trace included below.

In the following fe80::4432:4eff:fe51:1113 (aka 10.100.10.100) is the router,
(which is also the DNS/NAT/DHCP/... server) and fe80::3613:e8ff:fe2f:bbc4
(aka 10.100.10.151) is my laptop.  2001:470:a085:999::25 (aka 199.233.217.200)
is mail.netbsd.org of course.

Times are all UTC+07  (+/- 20 seconds... I don't bother running an NTP
client on my laptop, just ntpdate it every month or so, and it stays
"close enough")

Another 20 mins have passed since the "only 1 hour" above, and the default
v6 route is still absent...

jinx# route get -inet6 default
route: writing to routing socket: not in table

When I looked earlier:

jinx# route delete -inet6 default
delete net default
jinx# route get -inet6 default
   route to: ::
destination: ::
       mask: default
    gateway: fe80::4432:4eff:fe51:1113%iwm0
 local addr: fe80::3613:e8ff:fe2f:bbc4%iwm0
  interface: iwm0
      flags: 0x843<UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
       0         0         0         0         0         0         0         0 
jinx# tcpdump -i iwm0 -s 1600 -w /var/log/IWM0-DUMP

The initial "route delete" was yesterday, perhaps even the day
before, the "route get" was this morning, the tcpdump is what captued
the packets shown just below (the ssh happened from a non-root xterm).

kre

08:01:38.000997 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:01:44.861871 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:01:45.920148 IP 10.100.10.151.54189 > 10.100.10.100.53: 13156+ AAAA? mail.netbsd.org. (33)
08:01:46.145309 IP 10.100.10.100.53 > 10.100.10.151.54189: 13156 1/0/0 AAAA 2001:470:a085:999::25 (61)
08:01:46.145427 IP 10.100.10.151.54188 > 10.100.10.100.53: 28632+ A? mail.netbsd.org. (33)
08:01:46.376236 IP 10.100.10.100.53 > 10.100.10.151.54188: 28632 1/0/0 A 199.233.217.200 (49)
08:01:46.376367 IP6 fe80::3613:e8ff:fe2f:bbc4.64233 > 2001:470:a085:999::25.22: Flags [S], seq 950175903, win 32768, options [mss 1440,nop,wscale 3,sackOK,TS val 1 ecr 0], length 0
08:01:46.492703 IP6 fe80::4432:4eff:fe51:1113 > ff02::1:ff2f:bbc4: ICMP6, neighbor solicitation, who has fe80::3613:e8ff:fe2f:bbc4, length 32
08:01:46.492723 IP6 fe80::3613:e8ff:fe2f:bbc4 > fe80::4432:4eff:fe51:1113: ICMP6, neighbor advertisement, tgt is fe80::3613:e8ff:fe2f:bbc4, length 32
08:01:46.494232 IP6 fe80::4432:4eff:fe51:1113 > fe80::3613:e8ff:fe2f:bbc4: ICMP6, destination unreachable, unreachable route 2001:470:a085:999::25, length 88
08:01:52.384914 IP6 fe80::3613:e8ff:fe2f:bbc4.64233 > 2001:470:a085:999::25.22: Flags [S], seq 950175903, win 32768, options [mss 1440,nop,wscale 3,sackOK,TS val 13 ecr 0], length 0
08:01:52.386529 IP6 fe80::4432:4eff:fe51:1113 > fe80::3613:e8ff:fe2f:bbc4: ICMP6, destination unreachable, unreachable route 2001:470:a085:999::25, length 88
08:01:52.849432 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:02:02.679810 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:02:04.405197 IP6 fe80::3613:e8ff:fe2f:bbc4.64233 > 2001:470:a085:999::25.22: Flags [S], seq 950175903, win 32768, options [mss 1440,nop,wscale 3,sackOK,TS val 37 ecr 0], length 0
08:02:04.407008 IP6 fe80::4432:4eff:fe51:1113 > fe80::3613:e8ff:fe2f:bbc4: ICMP6, destination unreachable, unreachable route 2001:470:a085:999::25, length 88
08:02:20.600146 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:02:28.445815 IP6 fe80::3613:e8ff:fe2f:bbc4.64233 > 2001:470:a085:999::25.22: Flags [S], seq 950175903, win 32768, options [mss 1440,nop,wscale 3,sackOK,TS val 85 ecr 0], length 0
08:02:28.447575 IP6 fe80::4432:4eff:fe51:1113 > fe80::3613:e8ff:fe2f:bbc4: ICMP6, destination unreachable, unreachable route 2001:470:a085:999::25, length 88
08:02:29.611501 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:02:33.445243 IP6 fe80::4432:4eff:fe51:1113 > fe80::3613:e8ff:fe2f:bbc4: ICMP6, neighbor solicitation, who has fe80::3613:e8ff:fe2f:bbc4, length 32
08:02:33.445292 IP6 fe80::3613:e8ff:fe2f:bbc4 > fe80::4432:4eff:fe51:1113: ICMP6, neighbor advertisement, tgt is fe80::3613:e8ff:fe2f:bbc4, length 24
08:02:33.605204 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:02:41.490118 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:02:48.775761 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:02:53.565935 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:02:57.669666 IP6 fe80::4432:4eff:fe51:1113 > ff02::1: ICMP6, router advertisement, length 24
08:03:01.501772 IP 10.100.10.151.56761 > 199.233.217.200.22: Flags [S], seq 3492651494, win 32768, options [mss 1460,nop,wscale 3,sackOK,TS val 1 ecr 0], length 0
08:03:01.775397 IP 199.233.217.200.22 > 10.100.10.151.56761: Flags [S.], seq 422082190, ack 3492651495, win 32768, options [mss 1440,nop,wscale 3,sackOK,TS val 1 ecr 1], length 0
08:03:01.775419 IP 10.100.10.151.56761 > 199.233.217.200.22: Flags [.], ack 1, win 4197, options [nop,nop,TS val 1 ecr 1], length 0