in reply to Re: Batch processing of files
in thread Batch processing of files

Cleaned up code with error checking: 
use strict;
use warnings;

use Fcntl qw( SEEK_SET );

my @files = <files*.txt>;

# Should really use File::Temp
my $fn_out = "tmp.txt";

my $error = 0;
for my $fn_in (@files) {
    open(my $fh_out, '>', $fn_out)
        or die("Can't create output file \"$fn_out\": $!\n");

    open(my $fh_in, '<', $fn_in)
        or do {
            warn("Can't open input file \"$fn_in\": $!\n");
            $error = 1;
            next;
        };

    my $sum = 0;
    my $cnt = 0;

    while (<$fh_in>) {
        chomp;
        $sum += ( split /\|/ )[2];
        ++$cnt;
    }

    my $avg = $sum / $cti;

    seek($fh_in, 0, SEEK_SET)
        or do {
            warn("Can't seek in input file \"$fn_in\": $!\n");
            $error = 1;
            next;
        };

    while (<$fh_in>) {
        chomp;
        my ($hd, $md, $tl) = split(/\|/, $_);
        my $tlf = sprintf("%.4f", $tl/$avg);
        print $fh_out "$hd|$md|$tlf\n";
    }

    close($fh_in);
    close($fh_out)
        or do {
            warn("Can't save to output file \"$fn_out\": $!\n");
            $error = 1;
            next;
        };

    rename($fn_out, $fn_in)
        or do {
            warn("Can't rename \"$fn_out\" to \"$fn_in\": $!\n");
            $error = 1;
            next;
        };
}

exit($error);

[download]

Replies are listed 'Best First'.
Re^3: Batch processing of files
by austinby (Initiate) on Feb 11, 2010 at 19:14 UTC

    Thanks everyone.

    I think I found out what the problem was. Looks like I had to close all the sub-files, before populating the @files array.

    close(filehandle); #close newly generated files before array call.
my @files = <substituent_R*.txt>;

    [download]
    Austin-
[reply]
[d/l]

      Hi all,

      I just tested the code from ikegami in my script, and the last file was still not processed. I guess the key is, when I generate the sub-files separately, and then use this code, all the sub-files are processed. However, when include this code in the entire script, only the first 3 out of 4 files are processed. Is there a reason for this behaviour?

      I could include the entire script if that would help

      Thanks, Austin- 
      my @files = <substituent_R*.txt>;
my $fn_out = "tmp.txt";
my $error = 0;

for my $fn_in (@files) 
{
        open(my $fh_out, '>', $fn_out) or die("Can't create output fil
+e \"$fn_out\": $!\n");

        open(my $fh_in, '<', $fn_in) or do 
        {
                warn("Can't open input file \"$fn_in\": $!\n");
                $error = 1;
                next;
        };

        my $sum = 0;
        my $cnt = 0;

        while (<$fh_in>) 
        {
                chomp;
                $sum += ( split /\|/ )[2];
                ++$cnt;
        }

        my $avg = $sum / $cnt;

        seek($fh_in, 0, SEEK_SET)
        or do 
        {
                warn("Can't seek in input file \"$fn_in\": $!\n");
                $error = 1;
                next;
        };

        while (<$fh_in>) 
        {
                chomp;
                my ($hd, $md, $tl) = split(/\|/, $_);
                my $tlf = sprintf("%.4f", $tl/$avg);
                print $fh_out "$hd|$md|$tlf\n";
        }

        close($fh_in);
        close($fh_out) or do 
        {
                warn("Can't save to output file \"$fn_out\": $!\n");
                $error = 1;
                next;
        };

        rename($fn_out, $fn_in) or do
        {
                warn("Can't rename \"$fn_out\" to \"$fn_in\": $!\n");
                $error = 1;
                next;
        };
}

exit($error);

      [download]
[reply]
[d/l]

        However, when include this code in the entire script, only the first 3 out of 4 files are processed. Is there a reason for this behaviour?

        No. It's not the code. I see four possibilities:

        • There is no fourth file in @files.
        • It is being processed, but the average is very near 1.
        • An error is occurring and you're not noticing the error message.
        • You are not executing the code you think you are executing.

        After 

        my @files = <substituent_R*.txt>;

        [download]
        add 
        use Data::Dumper;
$Data::Dumper::Useqq = 1;
warn(Dumper(\@files));

        [download]

        Do you see the four files? Are there any weird characters?

        If you didn't answered yes and no, post the output and continue.

        After 

        my $avg = $sum / $cnt;

        [download]
        add 
        warn("Average=$avg\n");

        [download]

        Do you see four averages? Are any of them very near 1?

        If you didn't answered yes and no, post the output and continue.

        After every line, add 

        warn(__LINE__, " for $fn_in\n");

        [download]

        Post your output.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom