dear All, pls help me on below code this code is dealing with files having more than 1 million records 1st 6 line and last 2 lines are useless for me to process. major issue facing while writng files is data lost.i am working on multiple files. code: 
use POSIX;

$num_args = $#ARGV + 1;
if ($num_args != 1) {
  print "\nUsage: Spool.pl Require two argumrnts \n";
  exit;
}

@SMSFileList = `ls | grep CSV`;
chop(@SMSFileList);
my $FileCount = 0;

my %myhash;
my %myhash1;
my @lines;
my ($lines,$CDR,$uniquekey,$uniquekey1,$file_name,$file_name1,$CircleG
+roupHandle,$CircleGroupHandle1,$uniq1,$uniq2);
my $targetdir=$ARGV[0];
foreach my $SMSFileName (@SMSFileList)
{        $FileCount++;
     print "Reading File ($FileCount)\n";
     print "$SMSFileName\n";
         sysopen (SOURCE_SMS_FILE,"$SMSFileName",O_RDONLY) or die "Err
+or opening $SMSFileName";
         while(<SOURCE_SMS_FILE>)
    {     if ($_ =~ m/^,/)
     {
          @lines= split(",",$_);
                chop($_);
         if ($lines[14] =~ /^I$/ )
               {  
                   $uniquekey= $lines[17].$lines[14].$lines[21].$lines
+[44].substr($lines[27],0,8);
                   $CDR = substr($lines[27],0,8).','.$lines[28].','.$l
+ines[36].','.$lines[23].','.$lines[24].','.$lines[91].','.$lines[92].
+','.$lines[101].','.$lines[15].','.$lines[18].','.$lines[75].','.$lin
+es[21].','.$lines[44].','.$lines[14].','.substr($lines[69],0,8).','.$
+lines[1].','.$lines[13].','.$lines[50];
                                $file_name = $lines[17].'_'.$lines[14]
+.'_'.$lines[21].'_'.$lines[44].'_'.substr($lines[27],0,8);
                   $myHash{$uniquekey}++;
                   
                   }
             if($lines[14] =~ /^O$/ )
                   {
                   $uniquekey1= $lines[17].$lines[14].$lines[22].$line
+s[45].substr($lines[27],0,8);
                   $CDR1 = substr($lines[27],0,8).','.$lines[28].','.$
+lines[37].','.$lines[23].','.$lines[24].','.$lines[91].','.$lines[92]
+.','.$lines[101].','.$lines[16].','.$lines[19].','.$lines[75].','.$li
+nes[22].','.$lines[45].','.$lines[14].','.substr($lines[69],0,8).','.
+$lines[1].','.$lines[13].','.$lines[50];
        $file_name1 = $lines[17].'_'.$lines[14].'_'.$lines[22].'_'.$li
+nes[45].'_'.substr($lines[27],0,8);
                   $myHash1{$uniquekey1}++;
              }
       }
         foreach $uniquekey (keys %myHash)
        {
                $uniq1 = $file_name.'.csv';
                   sysopen($CircleGroupHandle,"$ARGV[0]/$uniq1",O_WRON
+LY|O_APPEND|O_CREAT)or die "Error writing to $!";          
                print $CircleGroupHandle "$CDR\n";
              # print "$uniquekey\n"
               #print "$key\n";
        }    
         %myHash = ();
         %$uniquekey = ();
         $uniquekey = {};
           
        foreach $uniquekey1 (keys %myHash1)
        {
               $uniq2 = $file_name1.'.csv';
               sysopen($CircleGroupHandle1,"$ARGV[0]/$uniq2",O_WRONLY|
+O_APPEND|O_CREAT)or die "Error writing to $!";  
               print $CircleGroupHandle1 "$CDR1\n";
             # print "$uniquekey1\n"
            
        }
        %myHash1 = ();    
        %$uniquekey1 = ();
        $uniquekey1 = {};
 }
close(SOURCE_SMS_FILE);
print"$SMSFileName closed\n";
close($CircleGroupHandle) or die "Error closing to $!";    
close($CircleGroupHandle1) or die "Error closing to $!"; 
print " All file handles closed\n"
}

[download]

In reply to <speed a major issue with below code also loosing data while writing in files> by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.