in reply to Re^2: Batch processing of files
in thread Batch processing of files

Thanks everyone.

I think I found out what the problem was. Looks like I had to close all the sub-files, before populating the @files array.

close(filehandle); #close newly generated files before array call. my @files = <substituent_R*.txt>;
Austin-

Replies are listed 'Best First'.
Re^4: Batch processing of files
by austinby (Initiate) on Feb 11, 2010 at 20:17 UTC

    Hi all,

    I just tested the code from ikegami in my script, and the last file was still not processed. I guess the key is, when I generate the sub-files separately, and then use this code, all the sub-files are processed. However, when include this code in the entire script, only the first 3 out of 4 files are processed. Is there a reason for this behaviour?

    I could include the entire script if that would help

    Thanks, Austin-
    my @files = <substituent_R*.txt>; my $fn_out = "tmp.txt"; my $error = 0; for my $fn_in (@files) { open(my $fh_out, '>', $fn_out) or die("Can't create output fil +e \"$fn_out\": $!\n"); open(my $fh_in, '<', $fn_in) or do { warn("Can't open input file \"$fn_in\": $!\n"); $error = 1; next; }; my $sum = 0; my $cnt = 0; while (<$fh_in>) { chomp; $sum += ( split /\|/ )[2]; ++$cnt; } my $avg = $sum / $cnt; seek($fh_in, 0, SEEK_SET) or do { warn("Can't seek in input file \"$fn_in\": $!\n"); $error = 1; next; }; while (<$fh_in>) { chomp; my ($hd, $md, $tl) = split(/\|/, $_); my $tlf = sprintf("%.4f", $tl/$avg); print $fh_out "$hd|$md|$tlf\n"; } close($fh_in); close($fh_out) or do { warn("Can't save to output file \"$fn_out\": $!\n"); $error = 1; next; }; rename($fn_out, $fn_in) or do { warn("Can't rename \"$fn_out\" to \"$fn_in\": $!\n"); $error = 1; next; }; } exit($error);

      However, when include this code in the entire script, only the first 3 out of 4 files are processed. Is there a reason for this behaviour?

      No. It's not the code. I see four possibilities:

      • There is no fourth file in @files.
      • It is being processed, but the average is very near 1.
      • An error is occurring and you're not noticing the error message.
      • You are not executing the code you think you are executing.

      After

      my @files = <substituent_R*.txt>;
      add
      use Data::Dumper; $Data::Dumper::Useqq = 1; warn(Dumper(\@files));

      Do you see the four files? Are there any weird characters?


      If you didn't answered yes and no, post the output and continue.

      After

      my $avg = $sum / $cnt;
      add
      warn("Average=$avg\n");

      Do you see four averages? Are any of them very near 1?


      If you didn't answered yes and no, post the output and continue.

      After every line, add

      warn(__LINE__, " for $fn_in\n");

      Post your output.

        Thanks. All the files were in the @files array, evidenced from a print statement, and none of the $avg was close to one. Closing the file handle after the foreach loop did the trick. I'll still experiment with the Data::Dumper module you suggested.

        Thanks!

        BTW, does anyone know how to create arrays dynamically? e.g.

        @files = ("a1.txt","a2.txt","a3.txt",...."an.txt");#n=50 foreach $file (@files) { my ($h1, $t1) = split(/\./,$file); #use the value of $h1 to create a new array @{$h1}; #to create arrays @a1, @a2, @a3, ....@an }