thank you pc88mxer and hipowls
I have moved it out of the loop and shaved off about close to 30 sec!!
grinder, I will also check your suggestion
All in all, I will just go back to developer and tell him that it might be just faster to run it on faster hardware

real    2m45.342s
user    2m43.562s
sys     0m1.684s

sub main {
      my $csv = Text::CSV_XS->new;
      for (@files) {
          open  ( NOW , "$directory/$_" ) || die "you suck\n";
           while (<NOW>) {
                  my (%rec,%HoH);
                  my $p;
                  chomp;
                  $t_counter++;

                  $csv->parse($_);
                  my @fields = $csv->fields;
[download]