The code block given to sort will not get any arguments (or, if it gets them, the arguments will be $a and $b. If you're not passing a code block as the first argument, you need to list all arguments separated by commas:

for (sort by_seq([qw(five nine one seven three)]), keys %h) {
[download]

(except that by_seq won't do what you want yet).

If you want to make the thing work, the easiest way is to generate the comparison subroutine on the fly and use it with sort:

use strict;
#use diagnostics;

sub by_seq { 
    my $seq_aref = shift;
    my %seq_helper; 
    { 
        my $index = 0; 
        for my $item (@{ $seq_aref }) { 
           $seq_helper{$item} = $index; $index++;
        };
    };

    # Here I return a subroutine that takes two arguments
    # (the left and the right element to be compared)
    return sub { $seq_helper{$_[0]} <=> $seq_helper{$_[1]} };
}; 

# -- 

my %h = (one=>2,three=>4,five=>6,seven=>8,nine=>0);

# Create our custom comparison routine
my $sorter = by_seq([qw(five nine one seven three)]);

for (sort {$sorter->($a,$b)}, keys %h) {
    print "$_: $h{$_}\n";
};
[download]

I thought I could write sort $sorter, LIST, but that gets interpreted by Perl (rightfully) as sort LIST, so I guess you'll be stuck with the extra code block.