It's running on an rPi4

Raspberry Pis don't have a Real-Time Clock with a backup battery, meaning on boot they have no idea what time it actually is, and they need to set the system time from an external source like an NTP server every time. time will change based on adjustments to the system time; if you're starting your script on boot or the system clock drifts and is adjusted by NTP during its runtime the measurement will be affected. If you want to know how long your script has actually been running in real-world time, independently of changes to the system clock, one way to do that is:

use Time::HiRes qw/clock_gettime CLOCK_MONOTONIC/;
my $start_time_s = clock_gettime(CLOCK_MONOTONIC);
...
my $run_time_s = clock_gettime(CLOCK_MONOTONIC) - $start_time_s;
[download]

See the documentation of clock_gettime for details - you may even want to use CLOCK_MONOTONIC_RAW depending on your requirements.

Edit: Better link to manpage