chiburashka,
My solution will likely not win the "fastest way" solution. There are likely CPAN modules that do this as well. Usually when someone wants the fastest way to do something - they are taking the wrong approach or using the wrong language. Perhaps if you don't get the answers you like you should expand on the overall picture.

#!/usr/bin/perl
use strict;
use warnings;

my @factors = Factor_It( 20 );
print "$_\n" for @factors;

sub Factor_It {
    my $number = shift;
    my @factors;
    for ( 2 .. int $number / 2 ) {        
        push @factors, $_ unless $number % $_;
    }
    return (1, @factors, $number);
}
[download]

Of course, you will also want to add some checks to ensure your input is a positive whole integer greater than 1. It looks like you updated your node after you posted it with the BigInt stuff - probably want a CPAN module with XS code. You can avoid checking even numbers if it is not evenly divisible by 2 on the first check