I assure you, I'm not changing the system time. ntpd could be getting involved, but that would only involve at most, tenths of seconds.

I explained that on a Raspberry Pi, NTP is most definitely involved (if connected to a network), and the time jumps are quite significant. Unless of course you've written your code in a way that it is not started on boot, but waits until the first time the NTP server is queried to start running, but you made no mention of that. You also mention "time of day, UnixUptime, and ^T value" but not the monotonic clock.