Suggestions as to what tests i can run are welcome.

The profiling you've done doesn't get into enough detail in the critical areas.

The first thing I would try, is isolating whether the extra time is spent reading from the file or shuffling memory. To that end, I'd see what happens to the timings if I just read the data but didn't store it:

#! perl -slw
#use 5.010;
use strict;
use Time::HiRes qw[ time ];;

sub x{
    open my $fh, '<', shift or die $!;
#    my @AoA;
    my $dummy = [ split ',' ] while <$fh>;
    close $fh;
    return $.;
}

for ( 1 .. 5 ) {
    my $start = time;
    printf "Records: %d in %.3f seconds\n",
        x( sprintf 'junk%d.dat', 1+ ($_ & 1) ),
        time() - $start;
}
[download]