Re: hardware timestamping of packets

[Vlad Balan <hbalan%usc.edu@localhost>]

This is a good question. I was planning to follow Linux's lead, where
the hardware counter and the system counter can be both reported, upon
request. For each cycle counter in the system Linux maintains a rough
translate to system time, approximating the skew and the offset based
on a few measurements and comparisons between the cycle counter and
system clock. This kind of timestamp translation is not hard to
implement.

I will start by providing support for the hardware timestamp and then
take care of translating the cycle counts to system time as well.

The rest of the discussion identified the problems present in
providing a socket interface for timestamping, on the transmission
path. I will try, for starters, to provide an interface that is
compatible with the Linux one in order to avoid too much
differentiation to userland applications. I agree that the current
form of the interface satisfies the needs of only a handful of
applications. I will try to maintain the implementation of
transmit-side timestamps clean in order to make it possible to
implement additional mechanisms for passing them on to userland.

I agree that providing hardware timestamps to bpf is a good idea. I
will try to see how to best implement it, given that bpf takes its
timestamps through a slightly different mechanism, close to the packet
arrival time. Here we might have to decide upon an interface again, to
specify the source of timestamps to be returned.

Finally, it appears that none of the cards currently available is
capable of obtaining hardware timestamps for all packets and returning
them in the packet descriptors. Line capture cards built on FPGAs have
this capability, however my understanding is that they use different
interfaces to the host (possibly userspace), making them unsuitable
for testing the hardware timestamping kernel feature.

Regards,
Vlad

On Wed, Jun 20, 2012 at 1:00 PM, Dennis Ferguson
<dennis.c.ferguson%gmail.com@localhost> wrote:
>
> On 12 Jun, 2012, at 23:06 , Vlad Balan wrote:
>> On Tue, Jun 12, 2012 at 7:36 PM, Darren Reed <darrenr%netbsd.org@localhost> 
>> wrote:
>>> On 13/06/2012 8:44 AM, Vlad Balan wrote:
>>> ...
>>>> First of all, most hardware cards that claim to do timestamping can
>>>> only timestamp a limited number of packets and use filters in order to
>>>> determine which packets to stamp.
>>> ...
>>>> When the filter is disabled and all packets are
>>>> timestamped, the registers are no longer locked and the timestamp
>>>> always corresponds to the latest received packet. Therefore if you
>>>> have multiple packets in one interrupt, you only get the hardware
>>>> timestamp of the last one. The only way to go around this (at least
>>>> for the Intel card) is to set a filter that makes most packet arrivals
>>>> create an interrupt, but then capturing at line speed would probably
>>>> not be feasible.
>>>
>>> What is the resolution of the time stamp?
>>> Microseconds? Nanoseconds? Picoseconds?
>>> Or is it in some other measure, such as ticks?
>>>
>>
>> The Intel card that I looked at returns values in 16ns increments. I
>> think the natural resolution would be nanoseconds
>
> Vlad,
>
> Have you managed to verify that 16 ns number with the actual hardware?
>
> The reason I'm asking is that the only reference I can find to 16 ns in
> the data sheet you linked to is in section 7.10.3.1, in the sentence that
> starts "For example if the cycle time is 16 ns and the incperiod is one...".
> That this isn't saying the actual hardware cycle time is 16 ns is suggested
> by the fact that the 82599 controller data sheet, here
>
>   
> http://www.intel.com/content/dam/doc/datasheet/82599-10-gbe-controller-datasheet.pdf
>
> duplicates that entire section (it is section 7.9.3.1 instead) with exactly
> the same 16 ns "For example" sentence, but follows it with a table showing
> what are apparently the actual hardware cycle times: 6.4 ns when the link is
> 10 Gbps, 64 ns when the link is 1 Gbps and 640 ns when the link is 100 Mbps.
> None of these match the 16 ns example, and the fact that the tick rate follows
> the link speed makes it seem (reading between the lines) like the clock may 
> not
> increment at all when there is no ethernet link.
>
> Unreliable, variable speed hardware counters don't make good time-of-day 
> clocks,
> so I'm interested in how you plan to use this and to present the results to
> consumers.  What will the timestamp value you return to the application look
> like, and how will the application interpret what that means?
>
> Dennis Ferguson