Get PTP configuration tied up and daemonized

[geerlingguy]

Right now I have a couple tasks in the ptp.yml tasks file that I still need to write:

Time Pi - Grandmaster

• Discipline the NIC's PHY clock (could take 2-4 minutes to get into the sub-100 ns range):
  • If using Chrony to sync the system clock from NTP/GPS: sudo phc2sys -s CLOCK_REALTIME -c eth1 -m -O 0
  • If using GPS directly: sudo ts2phc -c /dev/ptp0 -s generic --ts2phc.pin_index 2 --ts2phc.extts_polarity both --ts2phc.pulsewidth 100000000 -m -l 7
    • see Get PPS working with GPS #13 — and note the extts_polarity and pulsewidth would not be needed with the patch referenced in that issue)
    • Also note, there will be no PPS events until GPS position is locked
• Set up the Pi as a PTP grandmaster on eth1: sudo ptp4l -i eth1 -m --masterOnly 1 (could be --serverOnly on newer versions of linuxPTP)
  • Maybe set up configuration inside /etc/linuxptp/ptp4l.conf?
• Discipline the system clock from the NIC's PHY: sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME --step_threshold=1 --transportSpecific=1 -O 37 -m (NOTE: This offset is the current offset between TAI and UTC—it can change depending on leap seconds!)

Note: To wrap up the PTP configuration in automation, see this austinsnerdythings.com blog post, specificially the ptp4l config files).

Testing Sync on the Grandmaster:

Use testptp to set PPS out on the proper pin:

• Install with these instructions: Get PPS working with GPS #13 (comment)
• sudo testptp -d /dev/ptp0 -L0,2 (for the SMA closest to the 40-pin GPIO header)
• sudo testptp -d /dev/ptp0 -p 1000000000

Slave/Client Pi

This example assumes a Pi running Pi OS:

• Install linuxptp: sudo apt install -y linuxptp
• Stop timesyncd or any other preconfigured time sync service: sudo systemctl stop systemd-timesyncd
• Run sudo ptp4l -i eth0 --step_threshold=1 -m --slaveOnly 1 to begin time sync over PTP
  • You should quickly see a message like port 1: new foreign master 00a0c9.fffe.000002-1, and then the offset values should settle in < 100ns. Ideally...
• Discipline the Pi's internal clock from the PHY (could take 2-4 minutes to get into the sub-100 ns range):
  • sudo phc2sys -s eth0 -c CLOCK_REALTIME -q -m -w

To check if clocks are configured correctly and PTP is happy:

wget https://tsn.readthedocs.io/_downloads/f329e8dec804247b1dbb5835bd949e6f/check_clocks.c
gcc -o check_clocks check_clocks.c
sudo ./check_clocks -d eth1  # ethX depends on device setup
[see output here]

For reference:

$ phc2sys -h
usage: phc2sys [options]
 automatic configuration:
 -a             turn on autoconfiguration
 -r             synchronize system (realtime) clock
                repeat -r to consider it also as a time source

$ testptp -h
usage: testptp [options]
 -c         query the ptp clock's capabilities
 -d name    device to open
 -l         list the current pin configuration
 -L pin,val configure pin index 'pin' with function 'val'
            the channel index is taken from the '-i' option
            'val' specifies the auxiliary function:
            0 - none
            1 - external time stamp
            2 - periodic output
 -p val     enable output with a period of 'val' nanoseconds

[jauderho]

I was trying to use timemaster to set up both chrony and linuxptp but got sidetracked. Maybe this would be of interest to get both tied together?

[geerlingguy]

@jauderho - I was planning on configuring the individual portions by hand, but timemaster does seem like a potential way to tie it all together as well.

[geerlingguy]

Hmm... also https://github.com./MaciekMachni/ptp_recipes/blob/main/grandmaster.md for a tidy config (assuming NTP not running, in this case; see this NetDevConf presentation).

[geerlingguy]

Also just testing it manually.

In first session:

$ sudo ptp4l -i eth1 -m
ptp4l[242.573]: selected /dev/ptp0 as PTP clock
ptp4l[242.574]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[242.574]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[248.943]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[248.943]: selected local clock 00a0c9.fffe.000002 as best master
ptp4l[248.943]: port 1: assuming the grand master role

And second session:

# Download from https://github.com./Time-Appliances-Project/Incubation-Projects/tree/master/Software/testptp
$ cd testptp/
$ gcc -Wall -lrt testptp.c -o testptp
$ sudo mv testptp /usr/bin/

$ sudo testptp -d /dev/ptp0 -L2,0
set pin function okay

$ sudo testptp -d /dev/ptp0 -p 1000000000
PTP_PEROUT_REQUEST: Device or resource busy

[edit: This is still happening today if I try to run the same commands.]

[geerlingguy]

Interesting aside on phc2sys:

If I just run it unprivileged (phc2sys -a -rr), it exits immediately (seems like it should at least output a little message like 'insufficient privileges'. If I run it as sudo, it sits and doesn't seem to do anything if I try using auto config:

$ sudo phc2sys -a -rr
[sits forever...]

Instead of using auto config, I tried forcing it (on the grandmaster) to sync the eth1 PHC:

pi@time-pi:~/testptp $ sudo phc2sys -s eth1 -m -O 0
phc2sys[1468.085]: CLOCK_REALTIME phc offset -37000066486 s0 freq   +1339 delay   1833
phc2sys[1469.085]: CLOCK_REALTIME phc offset -37000069360 s1 freq   -1535 delay   1852
phc2sys[1470.085]: CLOCK_REALTIME phc offset     -1443 s2 freq   -2978 delay   1833
phc2sys[1471.085]: CLOCK_REALTIME phc offset        -5 s2 freq   -1972 delay   1852
phc2sys[1472.086]: CLOCK_REALTIME phc offset       434 s2 freq   -1535 delay   1870
phc2sys[1473.086]: CLOCK_REALTIME phc offset       426 s2 freq   -1413 delay   1852
phc2sys[1474.086]: CLOCK_REALTIME phc offset       308 s2 freq   -1403 delay   1851
phc2sys[1475.086]: CLOCK_REALTIME phc offset       179 s2 freq   -1440 delay   1834
phc2sys[1476.086]: CLOCK_REALTIME phc offset        88 s2 freq   -1477 delay   1852
phc2sys[1477.086]: CLOCK_REALTIME phc offset        31 s2 freq   -1507 delay   1852
phc2sys[1478.086]: CLOCK_REALTIME phc offset         1 s2 freq   -1528 delay   1870
phc2sys[1479.086]: CLOCK_REALTIME phc offset        -3 s2 freq   -1532 delay   1851
phc2sys[1480.087]: CLOCK_REALTIME phc offset         0 s2 freq   -1530 delay   1852
phc2sys[1481.087]: CLOCK_REALTIME phc offset         1 s2 freq   -1529 delay   1833
phc2sys[1482.087]: CLOCK_REALTIME phc offset        -2 s2 freq   -1531 delay   1833
...

[geerlingguy]

On a slave Pi:

pi@pi5:~ $ sudo ptp4l -i eth0 --step_threshold=1 -m
ptp4l[371.177]: selected /dev/ptp0 as PTP clock
ptp4l[371.178]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[371.178]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[372.803]: port 1: new foreign master 00a0c9.fffe.000002-1
ptp4l[376.803]: selected best master clock 00a0c9.fffe.000002
ptp4l[376.803]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[378.818]: master offset -37242662790 s0 freq      +0 path delay     44181
ptp4l[379.818]: master offset -37242711463 s1 freq  -48675 path delay     34814
ptp4l[380.818]: master offset     -11690 s2 freq  -60365 path delay     34814
ptp4l[380.818]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[381.818]: master offset       -445 s2 freq  -52627 path delay     25447
ptp4l[382.818]: master offset        640 s2 freq  -51675 path delay     18569
ptp4l[383.818]: master offset      -5929 s2 freq  -58052 path delay     18688
ptp4l[384.818]: master offset      -4605 s2 freq  -58507 path delay     16446
ptp4l[385.818]: master offset      -4405 s2 freq  -59688 path delay     16416
ptp4l[386.818]: master offset      -3206 s2 freq  -59811 path delay     16288
ptp4l[387.818]: master offset      -1815 s2 freq  -59382 path delay     16288
ptp4l[388.818]: master offset       -957 s2 freq  -59068 path delay     16288
ptp4l[389.818]: master offset       -258 s2 freq  -58656 path delay     16097
ptp4l[390.818]: master offset       -148 s2 freq  -58624 path delay     16085
ptp4l[391.818]: master offset        305 s2 freq  -58215 path delay     16085
ptp4l[392.818]: master offset       -140 s2 freq  -58569 path delay     16085
ptp4l[393.818]: master offset        -45 s2 freq  -58516 path delay     16029
ptp4l[394.818]: master offset       -211 s2 freq  -58695 path delay     16029

Letting it run a while, I see it settle in around 30-60ns sometimes, but then it gets jumpy up to 300 to -300 ns offset. Not the best :P

Using the check_clocks.c utility from the LinuxPTP docs, I get:

$ sudo ./check_clocks -d eth0
phc-rt delta is not 37 sec !
phc-tai delta is greater than 50 usec !
TAI offset set in kernel is not correct !
Please verify ptp4l and phc config and restart them if necessary to synchronize the clocks !

[geerlingguy]

On the grandmaster node:

$ sudo ./check_clocks -d eth1
phc-rt delta is not 37 sec !
phc-tai delta is greater than 50 usec !
Please verify ptp4l and phc config and restart them if necessary to synchronize the clocks !

[geerlingguy]

After working through #12 and #13, I have been able to get GPS PPS to the PHY, then used the PHY for Chrony (so it sends that time out over NTP), but I'm still dealing with three issues:

1. It seems like a kernel patch may be needed to get the rising-edge-only signal for ts2phc working (see Get PPS working with GPS #13)
2. (Possibly related to Get PTP configuration tied up and daemonized #1) The GPS time offset displays around 0.13 seconds after a while, as per cgps — though I might just need to record this data over time and manually adjust the offset.

I also need to wrap all the ptp4l/phc2sys/ts2phc configuration into files and systemd units. Great guide from @Nerdy-Austin here: https://austinsnerdythings.com/2025/02/18/nanosecond-accurate-ptp-server-grandmaster-and-client-tutorial-for-raspberry-pi/ — after that, also make sure to add a battery to the Pi's RTC battery header, so it can hold over at least using the little built in XO during an outage or reboot.

I also have had another read through @jclark's excellent ts2phc guide, and am happy to see a very tiny offset confirmed between the system clock and the PHC:

$ sudo phc_ctl eth1 cmp
phc_ctl[3323.900]: offset from CLOCK_REALTIME is -36999999973ns

$ chronyc tracking
Reference ID    : 50484330 (PHC0)
Stratum         : 1
Ref time (UTC)  : Thu Apr 10 18:23:35 2025
System time     : 0.000000012 seconds slow of NTP time
Last offset     : -0.000000017 seconds
RMS offset      : 0.000000012 seconds
Frequency       : 2.051 ppm slow
Residual freq   : -0.002 ppm
Skew            : 0.015 ppm
Root delay      : 0.000000001 seconds
Root dispersion : 0.000001002 seconds
Update interval : 1.0 seconds
Leap status     : Normal

[jclark]

My satpulse project https://satpulse.net is designed to make this a lot easier.

It can automatically handle NICs that timestamp both edges. First, it detects whether the NIC is one that does this. Second, if so, it can detect/configure the pulse width on u-blox receivers to distinguish the edges. If the pulse width is 0.1s and you have two pulses 0.1s apart, then you know the first one is the rising edge.

Getting your setup working with satpulse would involve:

• downloading and instaliing the .deb
• editing a couple of lines in the /etc/satpulse.toml config file (for serial port, serial speed, and network interface)
• using systemctl to start and enable the service

Hooking it up to chrony would involve:

• uncommenting a couple of lines in satpulse.toml
• adding a line to chrony.conf

I have this running on 7 different NIC/GPS combos and have some Ansible playbooks to automate the deployment and testing of whether it is working. It can write nice logs of the PHC/PPS offsets and there's a python script for analyzing those. The other checking mechanism is to have chrony setup to use both satpulse and several other NTP time servers (including a very accurate local one): if things go wrong (so satpulse doesn't match the others), chrony will use the other NTP servers, and so drop down to stratum 2.

[geerlingguy]

Working on daemonizing the config on the mini-time server, not on the one currently connected via Ethernet (mini-time is currently on WiFi so it can get good GPS reception).

When 'port 1' is not connected I get a FAULT_DETECTED, but I presume that will go away once I plug it in...

pi@mini-time:/etc $ systemctl status ptp4l
● ptp4l.service - Precision Time Protocol (PTP) service
     Loaded: loaded (/etc/systemd/system/ptp4l.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-04-10 16:01:01 CDT; 1s ago
   Main PID: 10683 (ptp4l)
      Tasks: 1 (limit: 4762)
        CPU: 2ms
     CGroup: /system.slice/ptp4l.service
             └─10683 /usr/sbin/ptp4l -f /etc/ptp4l.conf -i eth1

Apr 10 16:01:01 mini-time ptp4l[10683]: ptp4l[7805.414]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
Apr 10 16:01:01 mini-time ptp4l[10683]: ptp4l[7805.414]: port 1: link down
Apr 10 16:01:01 mini-time ptp4l[10683]: ptp4l[7805.414]: port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
Apr 10 16:01:01 mini-time ptp4l[10683]: [7805.414] port 0: INITIALIZING to LISTENING on INIT_COMPLETE
Apr 10 16:01:01 mini-time ptp4l[10683]: ptp4l[7805.414]: selected local clock 00a0c9.fffe.000002 as best master
Apr 10 16:01:01 mini-time ptp4l[10683]: ptp4l[7805.414]: port 1: assuming the grand master role
Apr 10 16:01:01 mini-time ptp4l[10683]: [7805.414] port 1: link down
Apr 10 16:01:01 mini-time ptp4l[10683]: [7805.414] port 1: LISTENING to FAULTY on FAULT_DETECTED (FT_UNSPECIFIED)
Apr 10 16:01:01 mini-time ptp4l[10683]: [7805.414] selected local clock 00a0c9.fffe.000002 as best master
Apr 10 16:01:01 mini-time ptp4l[10683]: [7805.414] port 1: assuming the grand master role

Edit: Testing that theory, I plugged in a cable and ptp4l showed:

Apr 10 16:03:41 mini-time ptp4l[10683]: [7965.398] port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
Apr 10 16:03:41 mini-time ptp4l[10683]: [7965.398] selected local clock 00a0c9.fffe.000002 as best master
Apr 10 16:03:41 mini-time ptp4l[10683]: [7965.398] port 1: assuming the grand master role

[geerlingguy]

I have almost everything wrapped into automation now, but I'm still missing a couple bits, I think.

I have a temporary antenna routed up into my ceiling area, so I'll let the whole thing chill overnight and see how things are synced in the morning with a 2nd Pi on my desk (which is going through three switches...).

[geerlingguy]

I'm also not syncing time of day... need to figure that out. Also saw the Tindie description is having same issue. (see below). Maybe need ptp4l v4?

TODO: Figure out how to get NMEA stream direct from GPS to work with ts2phc or into NIC PHC

[geerlingguy]

@jclark - Just now seeing your post on satpulse. I had a note somewhere to test it out... and probably will soon.

I'm finally reaching the point where I halfway understand how all these parts are working, so using satpulse would be useful in terms of seeing whether it has any of the same annoying issues I'm seeing (some that don't appear until after it's been going for an hour or two) with the i226.

I can adjust my playbook to use satpulse instead, at some point. United efforts definitely would keep things tidier :)

[geerlingguy]

I broke out the satpulse discussion to #16 — still want to get things wrapped up in here separately, then work on untangling everything once I have it working, and seeing if there are some things I still want to manage separately from Satpulse (or anything in my setup that could be added as an option in Satpulse...).