Re: ICMP_UNREACH_NEEDFRAG returns iface MTU instead of route?

[Dennis Ferguson <dennis.c.ferguson%gmail.com@localhost>]

On 21 Dec, 2013, at 17:55 , Dave Huang <khym%azeotrope.org@localhost> wrote:

> On Sat, Dec 21, 2013 at 03:03:48PM -0500, Dennis Ferguson wrote:
>> You can interpret it that way if you want, but that isn't what the text
>> really says.  The "MTU of the next-hop network" it is referring to is
>> the MTU of the network the outgoing interface is attached to; it is a
>> synonym for the next hop interface MTU.  This also means, of course, that
>> the text above doesn't really apply to your situation since "exceeds the MTU
>> of the next-hop network" is not the reason the datagram is being discarded
>> in your case.
> 
> How does the OS know what the MTU of the next-hop network is? By
> adding a static route, I'm declaring what I want the next hop's
> outgoing interface to be. It seems reasonable that I should also be
> able to use it to declare what the MTU is, if the default MTU assumed
> by the system based on the interface type is incorrect for whatever
> reason.

That's a good question.  Since a mismatch between the interface MTU and
another host's MRU causes painful problems, the standards try hard to make
sure this doesn't happen.  Often the MTU for a link type is a constant
defined in the standard which specifies how IP is mapped to the link.  When
it isn't a constant (e.g. point to point encapsulations, often) there is
often a protocol specified to ensure that everyone involved agrees
what the MTU should be (e.g. PPP).  In any case, if there's any possibility
that the choice of MTU the OS is making might not be the same as other
correctly operating hosts on the same link then the OS should also be
providing you with a way to correct its choice of interface MTU directly
by interface configuration.

You are right that you could effectively do the same thing with route
MTUs, if that's what a route MTU was for, but just being able to set an
MTU on a route yourself with `route add' is insufficient by itself.  Everything
which adds routes would need to know to do this, including interface
configuration (i.e. interface configuration still needs a correct MTU),
ARP and Neighbor Discover, DHCP, PPP and ICMP redirects.  A much better and
easier workaround for a bad interface MTU would be the ability to reconfigure
the interface MTU.

>> The reason is that the only
>> use I've seen made of the route MTU field is to implement a cache of PMTUs
>> discovered by applications running on the local host, and it is normally
>> considered quite inappropriate for a router to enforce the PMTUs learned
>> by its local applications on packets originated by other hosts.  It is
>> possible, however, that the field is no longer used for PMTU discovery
>> caching and that the remaining uses actually do require route MTUs to be
>> applied to forwarded packets for some reason.
> 
> It doesn't seem like NetBSD uses it as a PMTU cache--at least I've
> never seen "netstat -r" show any routes added due to PMTU discovery.

Here is a snippet from sys/netinet/tcp_output.c:

        size = tcp_mssdflt;
        if (tp->t_mtudisc && rt->rt_rmx.rmx_mtu != 0) {
                size = rt->rt_rmx.rmx_mtu - hdrlen;

TCP hence seems to think that when it is doing PMTU Discovery the learned
MTU will be stored in a route MTU.

Unfortunately there are more places where routes are stored than the structure
shown by "netstat -r".  The routes TCP looks at are apparently stored in the
"route cache" version of the table.  I don't know the command to display the
contents of that, but I do know that forwarding lookups also look in the
"route cache" version of the table so it isn't clear to me that the results
of PMTU Discovery aren't being applied to forwarded packets.

I really dislike the route cache thing, though that's a different topic.

> Later, when the ICMP needs frag is sent, the MTU sent back is the
> minimum of the interface MTU and the route MTU.

This is the problem with mechanisms which have ill-defined (or undefined)
purposes: everyone gets to have an opinion about how to use it, and there's
no reason for everyone to have the same opinion.

What I do know is that it is inappropriate for a router to enforce
PMTUs discovered by its local applications on packets originated by other
hosts, and that the route MTU on some routes seems to be coming from
PMTU Discovery.  I admit the argument that being bug-for-bug compatible
with Linux and/or FreeBSD is useful might have some merit, however.

Dennis Ferguson