Hi Monks,

I am creating a perl script which searches for a specific text in the CSV text file (100000 rows & 30 KB) and there are huge huge numbers of files. I am usign hashkeys to first put one file into the hash. And then search the specific text. After It finishes searches. I uses the same hash function to copy the second csv file of same size and search for a specific function.

The script ran perfectly for the 60 odds files but after that it crashes with "Out of Memory !".

While running script I am also observing from task manager the size of available memory continuously decreasing (2GB ram).

I think I am missing clearing the hash variable (@data1) and the the error message comes when my hach fully utilizes the full memory.

Question : How can I erase or clear the hash before my perl script takes the second file ? here is the sample code (shown only relevant code) 

# @ lines contain csv files

my %data1;
shift(@lines1); # remove column headings from file
shift(@lines1); # remove column headings from file


    foreach my $line (@lines1) {
        @words = split (/\,/, $line);            
        if ($words[6] > 90) {
        my $abstime = $words[1];
        my $payload = $words[5];
        $srcIPhex = substr $payload, 24, 8;
        my $dstIPhex = substr $payload, 32, 8;
        my $timestamp = substr $payload, 152, 12;            
        
        my $HashKey; # to get total number
        
        $HashKey = $srcIPhex.$abstime;        
        $data1{$HashKey}{ID} = $words[0];        
        $data1{$HashKey}{SRC_IP} = $srcIPhex;
        $data1{$HashKey}{DST_IP} = $dstIPhex;
        
        MeasureFiles(\%data1);
        
        }
                        
sub MeasureFiles
{
    my ($list_a_ref) = @_;        
  my %data1 = %$list_a_ref; # Dereference lists
  ....
  ....
  foreach (keys %data1) {                    
    $SrcIP_captured = inet_ntoa( pack( "N", hex( $data1{$_}{SRC_IP} ) 
+) );
    $DstIP_captured = inet_ntoa( pack( "N", hex( $data1{$_}{DST_IP} ) 
+) );
    foreach(my $i=0;$i<$ind;$i++){
       if ( $SrcIP_captured eq $SrcIP_ref[$i] &&  $DstIP_captured eq $
+DstIP_ref[$i]) {             
               $pkt_received++;                                       
+       
                   }
              }
          } 
         ....
         ....
    open(R1,">> $mainDirectory\\Results\\$file_result") || die("Cannot
+ Open File $file_result");
    my $results = "$SrcIP_ref[$i],$DstIP_ref[$i],$pkt_received";
    print R1 "$results\n";
    close(R1);
}

[download]

In reply to Perl script end up on saying "Out of Memory !" by syedumairali

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.