This (untested) code should do the trick. It uses a regex to extract the numbers and a GRT (see A brief tutorial on Perl's native sorting facilities.) to make the sort reasonably efficient.

## some.host.name [ip: 111.222.333.444] is alive  70.05 ms)
## Updated: switched 'N' for 'd'; real data.
my @sorted = map {
   unpack 'x[d] A*', $_
} sort map {
   my( $time ) = m[([\d.]+) ms)$];
   pack 'd A*', $time, $_;
} @data;
[download]

The regex will need tweaking if the latency can produce numbers in other units. Eg. whole seconds or microseconds instead of milliseconds.

You might need something like:

my( $time, $units ) = m[([\d.]+) (ms|us|secs)];
my $scaler = $units eq 'us' ? 0.1 : $units eq 'secs' ? 1000 : 1;
$time *= $scaler;
[download]