cybersmithuk has asked for the wisdom of the Perl Monks concerning the following question:

I have written a perl script which should run on 800 files in the same directory. Can anyone give me a clue pls.
  • Comment on How to execute the perl program on different files in the same directory

Replies are listed 'Best First'.
Re: How to execute the perl program on different files in the same directory
by Roy Johnson (Monsignor) on Apr 27, 2005 at 12:36 UTC
    Here is a little demonstration of the loop and file actions you need.
    for my $input_file (glob '*.txt') { (my $output_file = $input_file) =~ s/txt$/output/; open FILE, '<', $input_file or die "$!: $input_file\n"; # process input close FILE; open FILE, '>', $output_file or die "$!: $output_file\n"; # process output close FILE; }

    Caution: Contents may have been coded under pressure.
Re: How to execute the perl program on different files in the same directory
by Fletch (Bishop) on Apr 27, 2005 at 12:16 UTC

    perldoc perlvar, look for @ARGV. Alternately if you need to run on everything in a given directory, give the directory name as an argument and use opendir and readdir to build the list of files.

Re: How to execute the perl program on different files in the same directory
by hubb0r (Pilgrim) on Apr 27, 2005 at 13:16 UTC
    You could make yourself an array of files to work over using readdir and grep:
    opendir DIR, "Dir_With_Files" or die "Cannot open diretory: $!"; my @files = grep { /\.txt$/ } readdir DIR; closedir DIR; foreach my $file (@files) { do stuff; }
Re: How to execute the perl program on different files in the same directory
by Transient (Hermit) on Apr 27, 2005 at 12:06 UTC
    "I have written a Perl script which takes a single file as an argument and needs to be executed on 800 files in a single directory on <insert OS here>. My request is to have an easy way to run it on all files in that directory and/or to run it on each file individually. This is what I have tried: <insert tries here>."


    We don't know what you want. Please be more specific. I can give you about 100 answers that will answer your question, because it's not specific enough.
      This is my perl script which I need to run on all the files with .txt extension and create an output file individually with .output extension. The following script runs only on one file called "2.txt." If you can let me know what changes I need to do so that it will run on all the files in the directory with .txt and create an output file with .output. Thank you.
      #!/usr/bin/perl use warnings; # open a file and assign the filehandle FILE open(FILE, "2.txt") or die("can't open myfile.txt: $!\n"); while(<FILE>) { if($_ =~ /\s+(\S+)\s+(\S+)/) { $hash{$1} = $2; push(@index,$1); push (@a2, $2); } } close(FILE); # close the filehandle #print @a2, "\n"; #foreach$key(@index) #{ #print $key."====>".$hash{$key}."\n"; #} $i=0; open(FILEHANDLE, ">output.txt") or die "cannot open file for reading: +$!"; while ($i<= ($#a2 - 1)) { $c1=$a2[$i]; $c2=$a2[$i+1]; $c3=$a2[$i+2]; $c4=$a2[$i+3]; $c5=$a2[$i+4]; $i5 = $index[$i+4]; $c6=$a2[$i+5]; $c7=$a2[$i+6]; $c8=$a2[$i+7]; $c9=$a2[$i+8]; $i++; #print $c5, "\n"; if (($c1<$c2) && ($c2<$c3) && ($c3<$c4) && ($c4<$c5) && ($c5>$c6) && ( +$c6>$c7) && ($c7>$c8) && ($c8>$c9)) { $peak=$c5; #print $peak, "\n"; print FILEHANDLE "$i5 $peak\n"; } } close (FILEHANDLE);
        The following should have the makings of what you need (untested).

        I'm worried, though, that you are just plugging in stuff from books without trying to understand. For example, the following opens the file and then your 'die' message should be reporting which file could not be opened:

        open(FILE, "2.txt") or die("can't open myfile.txt: $!\n");

        Instead your code reports that some mystical 'myfile.txt' file could not be opened. The example below shows how the filename of the moment will appear in the error message if it has a problem opening.

        I haven't looked through the rest of your program, I assume it does what you want. The key element from my sample code is the opendir/readdir combination which lets you work with directories.

        Good luck with your quest of learning to use Perl!

        P.S. You will also want to learn a bit about how the 'markup' system works at perlmonks. I believe your [ and ] symbols are being interpreted as html links. Putting <code> in front of, and </code> at the end of your code should keep it from being mangled by the perlmonks input system. (UPDATE: I notice that your code has since been cleaned up, but still, understanding the perlmonks markup system can be challenging but worthwhile. :-)

        P.P.S. UPDATE: It occurred to me that since you are reusing the same code over many files, your use of global variables may get you in trouble. Each new file is going to be adding its contents into the variables %hash, @index, and @a2. You'd probably like them to start from scratch for each new file and so you should declare them anew with each pass through 'while'. Also, it appears that you will be blowing away the output with each

        open FILEHANDLE, "> output.txt"
        and so you should probably declare a new variable to hold the filename or plan on appending new info to the file with
        open FILEHANDLE, ">> output.txt"

        A way to force yourself to think about where variables should be declared and how long they should be in force ("in scope") is to put use strict; near the start of your code. You will find it imposes rules that keep you from doing things that get you into some forms of trouble. It will also be troubling, at first, to accomodate use strict;, but the effort will be worth it. I have attached a second version of my untested quickie code to illustrate the differences.

        $folder = '/tmp'; opendir TMPDIR, $folder or die "can't opendir $folder: $!\n"; while ( $filename = readdir TMPDIR ) { next unless $filename =~ /\.txt$/i; open FILE, "$filename" or die "can't open $filename: $!\n"; while(<FILE>) { if ($_ =~ /\s+(\S+)\s+(\S+)/) { $hash{$1} = $2; push(@index,$1); push (@a2, $2); } } close(FILE); # now do whatever } closedir TMPDIR;

        Second version:

        #!/usr/bin/perl -w # Even though W32 doesn't process the #! portion, I believe it # is smart enough to know that your inclusion of '-w' at the end # means that you want warnings reported. use strict; my $folder = '/tmp'; opendir TMPDIR, $folder or die "can't opendir $folder: $!\n"; while ( my $filename = readdir TMPDIR ) { next unless $filename =~ /\.txt$/i; my ( %hash, @index, @a2 ); open FILE, "$filename" or die "can't open '$folder/$filename': $!\n"; while(<FILE>) { if ($_ =~ /\s+(\S+)\s+(\S+)/) { $hash{$1} = $2; push(@index,$1); push (@a2, $2); } } close(FILE); # now do whatever } closedir TMPDIR;

        I suggest allowing a command line argument of the wildcard to process. You can do taint checking (i.e. check to make sure this user isn't being malicious)

        ARGV is an array of command-line arguments.

        You could take this and pass it to glob

        iterating on those results.

        That should get you started on the right path :)
Re: How to execute the perl program on different files in the same directory
by murugu (Curate) on Apr 27, 2005 at 12:16 UTC
    Hi cybersmithuk,

    Please specify what you are going to do so that it will be easier for us to answer.

    Regards,
    Murugesan
      I have a data on 800 files which are from HPLC machine which contains x and y axis. I would like to predict the peak points in those files for which I have written a script basing on one single file. I would like to implement the same script to all the files in the same directory at one go and store them in 800 output files as I hav further work to do on it. Hopefully, I have explained what exactly I am doing. Thank you very much for all your support for this new comer in Perl