Re: hardware timestamping of packets

[Matthew Mondor <mm_lists%pulsar-zone.net@localhost>]

Dennis Ferguson wrote:

> I think they return the entire transmitted packet to the error queue, so
> the contents of the packet are available to use to figure out which
> transmitted packet the timestamp corresponds to.

> I think the queue can be maintained like any datagram socket receive
> buffer; if you don't read it regularly it may fill to its limit, after
> which subsequent packets will be dropped off the end of the queue.
> I think this works.

> In contrast BSD kernels have managed to get by without this for other
> purposes so this would be a mechanism that would be added to (all?)
> sockets on the off chance that the application is going to want to
> timestamp outbound packets, something which I suspect very few
> processes are going to be interested in.
> Unless there is some sense that the "error queue" thing is going to be
> useful for a wider variety of applications than just this.  If
> timestamping packets is the only application for this it might be better
> to design a mechanism which only needs to exist when the application
> declares that it is going to be timestamping packets (perhaps using a
> setsockopt()/getsockopt() protocol).


Vlad Balan wrote:

> Therefore if you have multiple packets in one interrupt, you only get
> the hardware timestamp of the last one. The only way to go around this
> (at least for the Intel card) is to set a filter that makes most packet
> arrivals create an interrupt, but then capturing at line speed would
> probably not be feasible.


Thanks for the details,

It seems that indeed the simplest is to receive whole timestamped
packets back like on Linux (and probably good for an initial
implementation)... I looked at the Linux documentation for MSG_ERRQUEUE,
and have no opinion on if we'd want it in the future.  Are kqueue(2)'s
EV_ERROR a possible reason we don't need it?

But if we don't want to support it fully, it probably makes sense to
have such queue only be created on request, as Dennis suggested.
Either a setsockopt(2) IO_SNDTS to enable bidirectional mode,
or a setsockopt(2) to create a second socket and getsockopt(2) to
obtain its FD?

I guess that even if we wanted to support the general MSG_ERRQUEUE it'd
be possible to enable it on a per-socket basis using a setsockopt(2)
like SO_ERRQUEUE?

>  If some sort of "transaction ID" were included with the original
> sendmsg() then it wouldn't be required to return the entire
> transmitted packet to the application, just the timestamp and the
> "transaction ID" would do it.

> > It also makes the protocol synchronous, with two syscalls required per
> > sent packet.  I wonder if there's precedent on some other OSs for a
> > sendmsg(2) variant which can accept a  struct msghdr *?  If so, would
> > this be realistic using our current network+device stack to have the
> > status filled before the syscall returns?
> 
> I guess the difficulty with this is that the underlying transmit timestamp
> mechanism is actually asynchronous with respect to the sendmsg() syscall.
> That is, the syscall to send a datagram is generally finished when the packet
> is queued at the tail of the transmit queue of the output interface but the
> timestamp is unavailable from the hardware until after the packet is output
> to the wire.  There is a variable, and possibly quite large, time between
> these two events, so returning the result to a single system call is going
> to require a sleep to wait for completion where no sleep is done now.

Other potential ideas, which probably don't matter unless we need to
eventually obtain the timestamps of a steady fast flow of outgoing
packets, if a card supported that and the existing interface turned
out to be suboptimal:

If returning a timestamp for an outgoing packet before the syscall
returns is problematic, if we wanted to avoid whole packets, indeed a
transaction-id could be either user-provided via an ancillary message
(possibly problematic), or assigned by the kernel to packets via a
sendmsg(2)-variant, in which case a single read(2) could potentially
contain multiple id/timestamp entries...

A new kqueue(2) filter (say EVFILT_SENDMSGTS), but here again
transaction-IDs would have to fit in "ident", be process-wide-unique,
probably assigned via a sendmsg(2)-variant (say sendmsgts(2), and not
need EV_ADD), and timestamps would have to fit within the 64-bit "data"
field...

Some async signal with siginfo (I'm actually unsure of the efficiency
of this, and it has its own issues).  Perhaps by extension notifying
that an mmapped buffer (mapped read-only in the process) of
id/timestamp pairs is full, like is done for some DMA-capable devices
supporting mmap(2)...  but I think I'm getting in awful territory :)
-- 
Matt