my @sorted=map substr($_,4),
      sort map pack(N => (stat)[9]) . $_, @files;
[download]

Initially, to allow the maximum freedom on the actual size of an integer to hold a time specification, I would have liked to use the 'I' pack template as follows:

my $len=length pack I => 1;
my @sorted=map substr($_,$len),
      sort map pack(I => (stat)[9]) . $_, @files;
[download]

Unfortunately, these were not suitable for sorting ("at least" on my machine) due to little-endianness. Appearently there are no templates for "an unsigned integer of the same bit width as 'l' ones (at least 32) and with prescribed endiannes." I have been thinking about reverse, but that would make the code too clumsy too.

On a much simpler basis, I've also considered sprintf, but then if we look after GRT, it may mean we're concerned about efficiency. (Although in some cases like this one it can provide a very clean and terse solution too.) In which case I've run the following benchmark:

#!/usr/bin/perl

use strict;
use warnings;
use Benchmark qw/:all :hireswallclock/;

map substr($_,4),
  sort map pack(N => stat) . $_, @_;

my @files=glob '*';
my %time=map {$_ => (stat)[9]} @files;

sub spack { 
    map substr($_,4),
      sort map pack(N => $time{$_}) . $_, @_;
}

sub sprf1 { 
    map substr($_,10),
      sort map sprintf('%010u' => $time{$_}) . $_, @_;
}

sub sprf2 { 
    map substr($_,8),
      sort map sprintf('%08x' => $time{$_}) . $_, @_;
}

sub chk {
    my @a=(0,@_);
    shift @a <= $_ or return for @_;
    1;
}

my @sub=qw/spack sprf1 sprf2/;

for (@sub) {
    no strict 'refs';
    chk map $time{$_}, &{$_}(@files) or
      die "Error with sub '$_'\n";
}

cmpthese -30, { map { $_ => "$_(\@files)" } @sub };

__END__
[download]

(tested in a directory with ~1000 files) which gives me the following results:

          Rate sprf2 sprf1 spack
sprf2 756078/s    --   -7%  -24%
sprf1 811970/s    7%    --  -18%
spack 991485/s   31%   22%    --
[download]

Update: correct Benchmark results as per salva's remark. Funnily enough, the figures as far as percentages are concerned are not that different from the previous ones.

       Rate sprf2 sprf1 spack
sprf2 319/s    --   -6%  -21%
sprf1 340/s    6%    --  -16%
spack 403/s   26%   19%    --
[download]

YMMV