If the header line appears somewhere in the middle of the file, you could make use of a state variable (here $in_data_section). You'd initialize the variable to false, and set it to true as soon as you enconter the header line. And only when $in_data_section is true, you'd treat the input lines as data (numbers)...

(Similarly, if you can tell what ends your data section, you could then reset the state variable to false...)

...
my $in_data_section = 0;
while (<FILE>){
    chomp;
    if ($in_data_section) {
        my @numbers = split;
        #...
    }
    $in_data_section = 1 if m/^Number/;
}
[download]

Alternatively, if you know that there are exactly two data lines following the header, you could also do something like

while (<FILE>){
    chomp;
    if (/^Number/) {
        # found header, now we're expecting two lines with numbers
        for (1..2) {
            $_ = <FILE>;
            chomp;
            my @numbers = split;
            #...
        }
    }
    #... do other stuff
}
[download]

I really like jZed's DBD::AnyData for this sort of thing.