How do I process multiple files in parallel?

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: How do I process multiple files in parallel? by RazorbladeBidet (Friar) on Feb 25, 2005 at 15:44 UTC
`while (<>) { ... }` will process line by line, when you need what appears to be 3 lines at a time. Or you need to keep track of every set of three (perhaps using $.) Also, I don't suggest calling open inside the while loop. Update: Just noticed you're using $1, $2 and $3. For those three expressions, only $1 will be used, and overwritten each time. I suggest storing $1 in a variable until you're ready to use it after each expression, like: `/^Sen = (\S+)./; $sen = $1; /^Acc = (\S+)./; $acc = $1; /^Cor = (\S+)./; $cor = $1` [download] Updated as per merlyn's observation: `$sen = $1 if /^Sen = (\S+)./; $acc = $1 if /^Acc = (\S+)./; $cor = $1 if /^Cor = (\S+)./; push @array, $sen, $acc, $cor;` [download] But as mentioned before you'll have to do something different to get the array entries you want. -------------- It's sad that a family can be torn apart by such a such a simple thing as a pack of wild dogs	[reply] [d/l] [select]
Re^2: How do I process multiple files in parallel? by merlyn (Sage) on Feb 25, 2005 at 18:33 UTC
`/^Sen = (\S+).*/; $sen = $1;` [download] Never use $1 without testing the result of the match, or you'll be using a stale $1 from a previous match. -- Randal L. Schwartz, Perl hacker Be sure to read my standard disclaimer if this is a reply.	[reply] [d/l]
Re: How do I process multiple files in parallel? by holli (Abbot) on Feb 25, 2005 at 16:06 UTC
If you read x files at once, you will have to open and read from all of them: `use strict; use warnings; use IO::Handle; my @files = qw(t1.txt t2.txt t3.txt); my @handles = (); for ( @files ) { push @handles, new IO::Handle; open $handles[-1], "<", $_ or die "cannot open $_!\n"; } while (1) { my @data; for my $handle ( @handles ) { my $line = <$handle>; exit unless defined $line; chomp $line; $line =~ /= ([^=]+)/; push @data, $1; } print "@data\n"; }` [download] This script will safely exit when one of the files runs into eof. Update: I just noticed you need your data vertically: `use strict; use warnings; use IO::Handle; use Data::Dumper; my @files = qw(t1.txt t2.txt t3.txt); my @data = ([],[],[]); my @handles = (); for ( @files ) { push @handles, new IO::Handle; open $handles[-1], "<", $_ or die "cannot open $_!\n"; } EOF: while (1) { my $i=0; for my $handle ( @handles ) { my $line = <$handle>; last EOF unless defined $line; chomp $line; $line =~ /= ([^=]+)/; push @{$data[$i++]}, $1; } } print Dumper (\@data);` [download] The code works with n-files and n-lines per file. holli, /regexed monk/	[reply] [d/l] [select]
Re: How do I process multiple files in parallel? by trammell (Priest) on Feb 25, 2005 at 16:00 UTC
If you're just discarding the `Sen =` part of each line, you can simplify this to: `while (<>) { my ($foo) = /= (\S+)/; push @array, $foo; }` [download] Unless there's more to this problem...	[reply] [d/l]
Re: How do I process multiple files in parallel? by jdporter (Paladin) on Feb 25, 2005 at 17:04 UTC
Not that I'm deliberately trying to golf this thing, but... `undef $/; # so <> will sluurp each file on the command line into $_ $\ = "\n"; $, = " "; print /Sen = (\S+)/, /Acc = (\S+)/, /Cor = (\S+)/ while <>;` [download] Then redirect output when you call it.	[reply] [d/l]
Re: How do I process multiple files in parallel? by chas (Priest) on Feb 25, 2005 at 16:05 UTC
I would implement this as follows: First read all the filenames into an array, say @files. Next loop over the filenames: for each file create a string of form "x1 y1 z1" corresponding to the 3 values in that file (you could also use an array or hash, but a string will work; you can use matching to collect the values subject to the comments already made in a previous reply.) Save this string as an element of a global array, say @data. Finally, open your output file and print out the results from @data. (In your posted code, the opened file TEST doesn't seem to be used...maybe some of your code didn't display properly...) The actual code should be straightforward to write. chas	[reply]
Re: How do I process multiple files in parallel? by manav (Scribe) on Feb 25, 2005 at 16:07 UTC
- Move your open statement out of the while loop - are you writing the array into a file later?? - `open(OUT,"> TEST") or die($!: can't open output file") ; while(<DATA>) { push @array,$2 if (/^(Sen\|Acc\|Cor) = (\S+)/) ; } local $"="\n" ; print "@array" ;` [download] Manav	[reply] [d/l]
Re^2: How do I process multiple files in parallel? by Anonymous Monk on Feb 25, 2005 at 17:01 UTC
yes i will be	[reply]
Re: How do I process multiple files in parallel? by Sec (Monk) on Feb 25, 2005 at 16:09 UTC
As others commented, you should probably move the `open` out of the while loop. You also need to save the matched values in some variables. In your case I would do it like this: `open (OUT,">Test")\|\| die; while (<>){ /^Sen = (\S+)/ && do {$sen=$1}; /^Acc = (\S+)/ && do {$acc=$1}; /^Cor = (\S+)/ && do {$cor=$1; push @array,$sen,$acc,$cor; $sen=$acc=$cor=undef; }; }'` [download] Only if you encouter a "Cor" line (the last one of a block) handle the values you accumulated. And don't forget to clear them, just in case your input file is missing some lines.	[reply] [d/l] [select]
Re: How do I process multiple files in parallel? by Velaki (Chaplain) on Feb 25, 2005 at 16:33 UTC
Here's a cheesy way of doing this, -- with some test data -- especially if the data is small enough to slurp into the program whole. It uses regex from a different angle. `#!/usr/bin/perl use strict; use warnings; undef $/; $_ = <DATA>; print "$1 $2 $3\n" while /Sen = (\S+).?Acc = (\S+).?Cor = (\S+).*?/sg; __DATA__ Sen = x1 Acc = y1 Cor = z1 Sen = x2 Acc = y2 Cor = z2 Sen = x3 Acc = y3 Cor = z3 Filler Sen = x4 Acc = y4 Cor = z4` [download] Hope this helped, -v "Perl. There is no substitute."	[reply] [d/l]
Re^2: How do I process multiple files in parallel? by Anonymous Monk on Feb 25, 2005 at 16:36 UTC
most of the examples deal with printing the data vertically. i would like to present the information for each file horizontally in the TEST file	[reply]
Re: How do I process multiple files in parallel? by nobull (Friar) on Feb 27, 2005 at 10:25 UTC
There have been other good solutions already but as soon as I see something being done 3 times I immediately want to solve the problem in a way that easily exends to the general case. `my %record; while (<>) { $record{$1}=$2 if /^(\S+)\s=\s(\S+)/; if ( eof(ARGV) ) { print "@record{qw(Sen Acc Cor)}\n"; %record = (); } }` [download]	[reply] [d/l]
Re: How do I process multiple files in parallel? by g0n (Priest) on Feb 25, 2005 at 16:23 UTC
Deleted text - my answer was entirely incorrect. Profuse apologies; VGhpcyBtZXNzYWdlIGludGVudGlvbmFsbHkgcG9pbnRsZXNz	[reply]
Re^2: How do I process multiple files in parallel? by Anonymous Monk on Feb 25, 2005 at 16:30 UTC
that is correct, but the code i posted should read each file in the list?	[reply]
Re^3: How do I process multiple files in parallel? by g0n (Priest) on Feb 25, 2005 at 16:37 UTC
As per note above, text deleted. I was under the impression that passing the filename in would result in $_ containing only the filename, not the file content. I was wrong. My bad. Sorry. VGhpcyBtZXNzYWdlIGludGVudGlvbmFsbHkgcG9pbnRsZXNz	[reply]