I need help speeding up the following perl code. It pulls a line out of one file, splits into an array and uses one of the fields from the array as a filename to open a filehandle. It then splits this file into an array and searches for matches in the second file using variables taken from the first. Anyway the second search is taking is about 1 minute for every loop ( the files are around 180mb). I am using the foreach operator on this as I ma hoping to keep the file in memory whil doing the search ( there are up to 3000 searches per file) Any ideas gratefully accepted! Thanks, Phil 
#!/usr/bin/perl -w


$[ = 1;                 
$, = ',';               
$\ = "\n";              



if (! open CDR_DETAILS, "cdr_details.1") {
    die "couldn't open cdr_details.1: $! ";
    }

open CDR_TDM, ">cdr_tdm.csv";

$XDRFILE="DUMMY";

while (<CDR_DETAILS>) {
    chomp;
    @ITEM = split(',',$_,9999);
    $r++;
    print "Record number: $r";
    $i=0;
    $CDRDATE = $ITEM[1];
    print "CDRDATE: $CDRDATE";
    $CDRTIME = $ITEM[2];
    $CDRTIME =~ s/^0*//g;
    if ($CDRTIME le "0") { $CDRTIME="0"; }
    print "CDRTIME: $CDRTIME";
    $CLI1 = $ITEM[3];
    print "CLI1: $CLI1";
    $CLI2 = $ITEM[7];
    print "CLI2: $CLI2";
    if ( $ITEM[8] ne $XDRFILE) {
        print "$ITEM[8]!= $XDRFILE opening file!";
        $XDRFILE = $ITEM[8];
    
        if (! open XDR, "$XDRFILE") {
        die "couldn't open $XDRFILE :$!";
        }
         }
    
        
    
    foreach (<XDR>) {
             chomp;      
            @Fld = split(',', $_, 9);
    
            if ($Fld[2] eq ${CDRDATE} && $Fld[3] eq ${CDRTIME} && $Fld
+[5] eq ${CLI1} && $Fld[7] eq ${CLI2}) {
            select CDR_TDM;
        print $_;
        select STDOUT;
    
    }
}
    seek XDR,0,0;
}

[download]

In reply to Should this be so slow ? by phi.jones

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.