In addition to what you said, there's also the problem that the output the OP wants likely is from the bash builtin "time".  Perl, however, is typically configured to call /bin/sh as the default shell (see perl -V:sh). And as sh doesn't know the time builtin (even if /bin/sh is just a link to /bin/bash — for compatibility reasons, the program name is what matters here), the shell will search for an external executable "time", which usually is /usr/bin/time. The latter produces a different output format, though.

To sum up, what the OP probably wants is

my $times = `/bin/bash -c "(time ls) 2>&1 >/dev/null"`;
[download]

Compare

#!/usr/bin/perl -l

$times = `(time ls) 2>&1 >/dev/null`;
print "/usr/bin/time:\n[$times]";

$times = `/bin/bash -c "(time ls) 2>&1 >/dev/null"`;
print "bash builtin:\n[$times]";

__END__
/usr/bin/time:
[0.00user 0.00system 0:00.00elapsed 114%CPU (0avgtext+0avgdata 0maxres
+ident)k
0inputs+0outputs (0major+396minor)pagefaults 0swaps
]
bash builtin:
[
real    0m0.008s
user    0m0.004s
sys     0m0.004s
]
[download]