in reply to Re^3: Batch processing of files
in thread Batch processing of files

Hi all,

I just tested the code from ikegami in my script, and the last file was still not processed. I guess the key is, when I generate the sub-files separately, and then use this code, all the sub-files are processed. However, when include this code in the entire script, only the first 3 out of 4 files are processed. Is there a reason for this behaviour?

I could include the entire script if that would help

Thanks, Austin-
my @files = <substituent_R*.txt>; my $fn_out = "tmp.txt"; my $error = 0; for my $fn_in (@files) { open(my $fh_out, '>', $fn_out) or die("Can't create output fil +e \"$fn_out\": $!\n"); open(my $fh_in, '<', $fn_in) or do { warn("Can't open input file \"$fn_in\": $!\n"); $error = 1; next; }; my $sum = 0; my $cnt = 0; while (<$fh_in>) { chomp; $sum += ( split /\|/ )[2]; ++$cnt; } my $avg = $sum / $cnt; seek($fh_in, 0, SEEK_SET) or do { warn("Can't seek in input file \"$fn_in\": $!\n"); $error = 1; next; }; while (<$fh_in>) { chomp; my ($hd, $md, $tl) = split(/\|/, $_); my $tlf = sprintf("%.4f", $tl/$avg); print $fh_out "$hd|$md|$tlf\n"; } close($fh_in); close($fh_out) or do { warn("Can't save to output file \"$fn_out\": $!\n"); $error = 1; next; }; rename($fn_out, $fn_in) or do { warn("Can't rename \"$fn_out\" to \"$fn_in\": $!\n"); $error = 1; next; }; } exit($error);

Replies are listed 'Best First'.
Re^5: Batch processing of files
by ikegami (Patriarch) on Feb 11, 2010 at 22:35 UTC

    However, when include this code in the entire script, only the first 3 out of 4 files are processed. Is there a reason for this behaviour?

    No. It's not the code. I see four possibilities:

    • There is no fourth file in @files.
    • It is being processed, but the average is very near 1.
    • An error is occurring and you're not noticing the error message.
    • You are not executing the code you think you are executing.

    After

    my @files = <substituent_R*.txt>;
    add
    use Data::Dumper; $Data::Dumper::Useqq = 1; warn(Dumper(\@files));

    Do you see the four files? Are there any weird characters?


    If you didn't answered yes and no, post the output and continue.

    After

    my $avg = $sum / $cnt;
    add
    warn("Average=$avg\n");

    Do you see four averages? Are any of them very near 1?


    If you didn't answered yes and no, post the output and continue.

    After every line, add

    warn(__LINE__, " for $fn_in\n");

    Post your output.

      Thanks. All the files were in the @files array, evidenced from a print statement, and none of the $avg was close to one. Closing the file handle after the foreach loop did the trick. I'll still experiment with the Data::Dumper module you suggested.

      Thanks!

      BTW, does anyone know how to create arrays dynamically? e.g.

      @files = ("a1.txt","a2.txt","a3.txt",...."an.txt");#n=50 foreach $file (@files) { my ($h1, $t1) = split(/\./,$file); #use the value of $h1 to create a new array @{$h1}; #to create arrays @a1, @a2, @a3, ....@an }