in reply to Script not working on large file size

Without seeing your script, your description of the problem is inadequate for us to provide a diagnosis.

There is the possibility that you're slurping the file, which doesn't scale well.


Dave

  • Comment on Re: Script not working on large file size

Replies are listed 'Best First'.
Re: Script not working on large file size
by bh_perl (Monk) on Apr 22, 2004 at 11:23 UTC
    Ok... That is my script
    #!/usr/bin/perl -w use Fcntl; use Getopt::Std; use constant DIR => "/d/testing"; use constant MAX => 11; my $prog = $0; $prog =~ s|.*/||; my %opts; getopts('o:', \%opts); my $cmd = $opts{o} || 1; my @date = localtime(time); my $dd = sprintf ("%02d", $date[3]); my $mm = sprintf ("%02d", $date[4] + 1); my $yy = sprintf ("20%02d", $date[5] % 100); my $yymmdd = "$yy$mm$dd"; my @a; my $bnum; my $template = "A12 A4 A20 A20 A2 A2 A2 A2 A12 A4 A24 A24"; unshift (@ARGV, "-") unless @ARGV; for my $mfile (@ARGV) { $file = $mfile; open (IN, "< $file") or die ("Can't open $file: $!\n"); while (<IN>) { @normalDb = (); @errorDb = (); s/\s//g; $dd = $_; my @a = unpack($template, $dd); $sdate = $a[8]; $state = $a[4]; $err = $a[7]; $bnum = substr($a[3], skip_zero($a[3])); substr($dd, 37, 0) = '00' if ( $state eq '02' ); substr($dd, 62, 3) = '01' if ( $sdate !~ /^0/ && $err !~ /^0/ + ); substr($dd, 62, 3) = '01' if (length($dd) == 131); #The 2 last character of lines must be number if ($dd !~ /\d\d$/ || $dd =~ /<u$/ ) { $dd =~ s/..$/00/g; } #insert X for the rest if ( length($dd) < 130 ) { $dd = insert0($dd); } else { $dd = substr($dd, 0, 130); } if ( length($dd) == 130 ) { push (@normalDb, $dd); } else { push (@errorDb, $dd); } if ( @normalDb > 0 ) { my $path = "$file\.ok"; open (NORMAL, "+>> $path") or die $!; for (@normalDb) { print (NORMAL "$_\n") }; close (NORMAL); } if (@errorDb > 0 ) { my $path = "$file\.error"; open(ERROR, "+>> $path") or die $!; for (@errorDb) { print (ERROR "$_\n") }; close (ERROR); } } close (IN); } sub skip_zero { my ($a) = @_; my $j = 0; foreach my $i (split //, $a) { $j++; return $j if ($i > 0); } } sub insert0 { my ($tmp) = @_; my @data; push(@data, $tmp); for (my $i = length($tmp); $i < 130; $i++) { push(@data, "X"); } return sprintf ("%s", join('', @data)); } die;
    As your info, the the file size is about 507156496 bytes....
      ...its working but not on larger files ?

      Nothing stands out as particulalry wrong with your code from a quick browse--except maybe that it is probably quite slow. However, you don't say in what way it is not working on large files? Does it produce error messages? Give the wrong output? Never finish?

      Are you simply not waiting long enough for it to complete?

      Without some hints as to how it's failing, working out why is hard.


      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "Think for yourself!" - Abigail
        Thanks,
        I spend about 2/3 hour to monitor the script and its done whitout any error message. But, some of the data are not changes as per requirements. As example, the 2 last character of lines must be integer, if not then changes to default values ("00")
        #from my script ... if ($dd !~ /\d\d$/ || $dd =~ /<u$/ ) { $dd =~ s/..$/00/g; } ...
        But, the script skips on that case and once I test again on small files, it done..