In the Linux world, many drivers are written for Network Interface Cards (NICs) that support Hardware timestamping. The Linux application used for PTP is usually ptp4l that comes in the apt-package “linuxptp”. In the ideal case when one runs ptp4l on two hosts (Linux machines) with interconnected NICs, the applications show us the accuracy at which both the NICs’ hardware clocks are in sync. As depicted below.
PTP is a widely used Protocol to bring Time sensitivity in Ethernet devices that rely on serial data transfers. The 1588 Standard paves the way to synchronize clocks of multiple hosts with a Grandmaster Clock. Timestamped PTP packet exchanges achieve the same. Hardware support for the timestamping is quintessential.
Figure 1: Example ptp4l application behaviour.
Sometimes, as developers, this is not the case and we observe issues in getting the clocks to sync. It is important to note that although clocks are used to sync date and time, PTP defines the exchange of time in Seconds and nanoseconds only. The same is used in PTP packets. The epoch time for Zero Seconds and Zero nanoseconds is 1-1-1970 5:30 AM IST. The basic packets that are exchanged as a part of PTP are
Sometimes, as developers, this is not the case and we observe issues in getting the clocks to sync. It is important to note that although clocks are used to sync date and time, PTP defines the exchange of time in Seconds and nanoseconds only. The same is used in PTP packets. The epoch time for Zero Seconds and Zero nanoseconds is 1-1-1970 5:30 AM IST. The basic packets that are exchanged as a part of PTP are
- Announce packet:
- Sent by the NIC assuming the Grand master role.
- Sync packet
- Sent by the Grand master clock.
- The time when this packet is transmitted is captured.
- Follow up packet
- Sent by the Grandmaster to inform the slave clocks as to when in Grandmaster time was the sync packet was sent.
- Essentially, puts the sync packet’s timestamp in this packet and transmits
- This packet is used by the slave to correct clock difference up to seconds’ level.
- Delay request packet
- Sent by the slave clock.
- The timestamp of this packets’ exit is captured by the slave.
- These packets are to correct nanoseconds level clock differences due to path delay between the Grandmaster and the slave.
- Delay response packet
- Sent by the Grandmaster.
- Contains the time at which the Grandmaster received the Delay request packet.
Now that we have laid the groundwork, Let’s see what endpoints (function pointers/ APIs) Linux provides us with to implement, for the NICs driver.
The ptp_clock_info struct holds the function pointers to the APIs used by the slave clock for PTP flow as described above, listed as follows:
- adjtime()
- Is called in the first few sync cycles to accommodate for the difference in time up to seconds’ level.
- adjfreq()
- Is called to narrow down time up to nanoseconds level by adjusting the clock sampling frequency in proportions to the physical Oscillator frequency, depending on the required nanosecond accuracy.
skb_shared_hwtstamps structure defined in Linux is supposed to be filled with the total time in nanoseconds (ie, seconds * 109 + nanoseconds) of the snapshot taken by the hardware for the transmitted or received packet. The snapshots taken by the hardware are usually available in registers or Descriptors. This is the point at which timestamps are handed over to the stack for PTP related calculations.
When ptp4l receives HW timestamps for the packets it generated or received, It makes a call to the adjtime() API, where we bring in the hardware clocks to sync at about seconds range. This usually happens after 2 sync cycles. The master offset value seen Figure 1 is passed as an argument to the adjtime() API. Once the clocks are in sync up-to seconds, the ptp4l keeps calling the adjfreq() to keep the clocks in sync in the nanoseconds range. The frequency offset in the image above is passed as a parameter to the adjfreq() call. If the NICs are connected back-to-back a master offset of less than 100ns can be seen for a prolonged duration (agin depends on the required nanoseconds accuracy).
The ptp4l showed an out-of-sync-master slave with maxed-out frequency offset. In our case, it was a bug in the conversion of time in seconds and nanoseconds register, to complete nanoseconds only. Usually, hardware maintains two separate registers or descriptor fields of 32 bits each. The stack provides a 64-bit nanoseconds field for the overall timestamp. If the conversion is wrong, we would be misinforming the Linux stack. In this case, the adjtime() called would not have corrected the slave clock to the master clock up-to seconds level at all. This forces the adjfreq() to max out on the slave clock frequency offset, yet not syncing time. This usually gives out a timeout error on the ptp4l application log.
We could have corrected this sooner if we had approached the problem from a timestamp handover point rather than from a device perspective.
Hence, beginning the debug at the point where timestamps are handed over to the Linux stack is a good starting point in the debugging process. One can also trace out the PTP packets and their timestamps through tools like Wireshark if the problem did not lie in the timestamp handover. If the problem still persists, it should be looked for in the adjfreq() API and device-specific details.
