For efficiency purposes, the trap to avoid is allowing sort to do a -M on the same file each time it lands on that file during the sorting algorithm - the ||= operator can be useful in this regard...

my %filedate;
my @lowest_priority =
  sort {
    ( $filedate{ $a } ||= -M $a )
    <=>
    ( $filedate{ $b } ||= -M $b )
  } grep /^Flat_file.*\.txt$/, readdir LOWEST;
[download]