How about unpacking in chunks of 3 byte resulting in 4 6-bit numbers.

Then create a few tables beforehand to remove the bit shift operations. For example the second 6-bit number would consist of the last two bit of the first byte and the first 4 bit of the second byte. So:

my @table2, @table3;
foreach my $i (0..256) { 
  $table2[$i]= $i >> 6;
  $table3[$i]= $i << 2 & 63;
}

...
my $number[0]= $byte[0] & 63;
my $number[1]= $table2[$byte[0]] + $table3[$byte[1]];
...
[download]

UDPATE: Naturally it should be profiled whether the table lookup operation is really faster than a bit shift plus bit and