Timekeeping is messy; PC hardware sucks with errors >100 ppm not uncommon. You can configure ntpd to not discipline the kernel clock. Use adjtimex to manually check and compensate for drift, ntpdate to set the time. Or just enjoy the bumpy ride, network hiccups etc....