weird IPv6 packet dropping with 6to4

[Taylor R Campbell <campbell%mumble.net@localhost>]

[I am not subscribed to this list; please cc me in replies.]

I have a home gateway running NetBSD 5.0_STABLE.  This machine has a
public IPv4 address and talks with my ISP using PPPoE.  I am using
pkgsrc's net/hf6to4 to configure it as a 6to4 router.  When I point my
web browser at web sites whose hostnames have AAAA records, such as
NetBSD.org and ietf.org, so that my web browser tries to talk the HTTP
over IPv6, some packets are dropped en route through my gateway from
the remote server to my local laptop, causing the web pages not to
load, or to load extremely slowly.

I have collected three packet traces during such an HTTP transaction,
one on the external interface, one on the stf0 interface, and one on
the internal interface.  These are the output of `tcpdump -w', and are
provisionally at

<http://mumble.net/~campbell/tmp/tcpdump/fxp0.dump>  # external
<http://mumble.net/~campbell/tmp/tcpdump/stf0.dump>  # 6to4
<http://mumble.net/~campbell/tmp/tcpdump/ex0.dump>   # internal

Note that fxp0 and stf0 see the last few packets addressed from the
server to my web browser, but ex0 does not see them.

I have puzzled this over thoroughly, and have no further clues, so I
thought I'd put the question to netbsd-users@ of what might be going
on here and how I can fix it.  Please let me know if you'd like any
further information about my configuration.

Here is the output of `ifconfig -a' on the gateway; fxp0 is the
external ethernet interface, attached to the DSL modem, and ex0 is the
internal ethernet interface, attached to my LAN.

fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        address: 00:90:27:a8:7e:60
        media: Ethernet autoselect (100baseTX 
full-duplex,flowcontrol,rxpause,txpause)
        status: active
        inet6 fe80::290:27ff:fea8:7e60%fxp0 prefixlen 64 scopeid 0x1
ex0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        
capabilities=3f00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx>
        
enabled=3f00<IP4CSUM_Rx,IP4CSUM_Tx,TCP4CSUM_Rx,TCP4CSUM_Tx,UDP4CSUM_Rx,UDP4CSUM_Tx>
        address: 00:b0:d0:80:38:c3
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
        inet6 fe80::2b0:d0ff:fe80:38c3%ex0 prefixlen 64 scopeid 0x2
        inet6 2002:8d9a:cf6e:2::1 prefixlen 64
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33192
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
pppoe0: flags=8851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST> mtu 1492
        inet 141.154.207.110 -> 10.9.95.1 netmask 0xff000000
        inet6 fe80::290:27ff:fea8:7e60%pppoe0 ->  prefixlen 64 scopeid 0x4
stf0: flags=1<UP> mtu 1280
        inet6 2002:8d9a:cf6e:1::1 prefixlen 16

I am using pf to filter packets and to provide IPv4 NAT:

# /sbin/pfctl -s nat
nat on pppoe0 inet from 10.0.1.0/24 to any -> (pppoe0) round-robin
# /sbin/pfctl -s rules
scrub in on ex0 all fragment reassemble
scrub out on pppoe0 all max-mss 1440 fragment reassemble
scrub on stf0 all max-mss 1024 fragment reassemble
block drop all
pass quick on lo0 all flags S/SA keep state
pass quick on pppoe0 inet proto ipv6 all keep state
pass inet proto icmp all keep state
block drop in quick on pppoe0 from any to (ex0:network)
pass in on pppoe0 inet proto tcp from any to (pppoe0) port = ssh flags S/SA 
keep state (source-track rule, max-src-conn 10, max-src-conn-rate 5/60, 
src.track 60)
pass in on ex0 from (ex0:network) to any flags S/SA keep state
pass out on ex0 from any to (ex0:network) flags S/SA keep state
pass in on pppoe0 inet proto udp from any to (pppoe0) port = domain keep state
pass out on pppoe0 proto tcp all flags S/SA modulate state
pass out on pppoe0 proto udp all keep state
pass out on pppoe0 proto icmp all keep state
pass quick on stf0 inet6 all flags S/SA keep state
pass on ex0 inet6 proto ipv6-icmp all keep state

However, I have also tried eliminating all my pf rules except for the
nat rule and `pass quick all', and that didn't solve the problem, so I
don't think that my pf rules are the source of the problem -- except
perhaps those related to the MSS.

As you can see, I have set up the usual MSS-clamping for PPPoE, and
for IPv6 packets going through stf0, I have imposed absurdly small
MSS-clamping, to ensure that it is not at issue.  I also tried 1200,
rather than 1024, and there was little significant difference between
the two.  However, before I added any `scrub on stf0 all max-mss ...',
the IPv6-in-IPv4 encapsulated packets were being fragmented (at the
IPv4 level, of course), but that didn't change whether they got passed
along to ex0.  For comparison, here are three analogous dumps, with
the scrubbing rule omitted, and thus MSS values that are probably
wrong:

<http://mumble.net/~campbell/tmp/tcpdump/fxp0mss.dump>  # external
<http://mumble.net/~campbell/tmp/tcpdump/stf0mss.dump>  # 6to4
<http://mumble.net/~campbell/tmp/tcpdump/ex0mss.dump>   # internal

Here is the content of my hf6to4.conf:

out_if="pppoe0"
in_if="ex0"
v6_net="1"
v6_innernet="2"
v6_prefixlen=16
hostbits6=":1"
peer="6to4-anycast"