Something like this perhaps:

#!/usr/bin/perl

use strict;
use warnings;

my $end_time = "16:54:28";
my $start_time = "16:54:23";

my @start = split /:/, $start_time;
my @end   = split /:/, $end_time;

my @diff;

$diff[$_] = $end[$_] - $start[$_] for 0 .. $#start;

if ($diff[2] < 0) {
  $diff[1]--;
  $diff[2] += 60;
}

if ($diff[1] < 0) {
  $diff[0]--;
  $diff[1] += 60;
}

print "@diff\n";
[download]

This ends with @diff containing the difference in hours, minutes and seconds. You can format that in any way you want.