If you define a number as anything containing an optional +/-, one or more digits, and an (optional) period, then you could simply do something like:

m/([+\-]?[0-9\.]+)/g;
[download]

Using your example, and putting this into a short script:

#!/usr/bin/perl -w
use strict;
use Data::Dumper::Simple;

my @numbers;

while (<DATA>) {
    @numbers = $_ =~ m/([+\-]?[0-9\.]+)/g;
}

print Dumper(@numbers);

__DATA__
iters = 320 (3.05s) aver = 9568.34 us (9603.45 us, 1.01s, 99.79%)
[download]

Which gives:

@numbers = (
             '320',
             '3.05',
             '9568.34',
             '9603.45',
             '1.01',
             '99.79'
           );
[download]

Does this help?

Cheers,
Darren :)