Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

HI, i just wondered if anyone could help, i am trying to write a shell script that will run 5 of my programs in sequential order. i know how to do this, however i don't know how to do it so that the input files can be different and i also want to pipe the out put file so that it can be the input file of the next program. e.g 
at the minute i have it like this;

echo "this is a shell script"

perl ./program1 prog1.input prog1.output
perl ./program2 prog2.input prog2.output

etc

[download]
however,the output of prog1 is the input of prog2 etc. ideally i don't want the file names to matter, i would like for anyone to use my programs. thanks, i realise this isn't strictly a perl question.

Replies are listed 'Best First'.
Re: UNIX shell scripts + pipes
by Kanji (Parson) on May 27, 2002 at 14:57 UTC

    Without changing anything, you can use "-" as your input and output filenames when piping between scripts, which open inteprets as wanting to use STDIN or STDOUT depending on context.

      perl ./program1 prog1.input - \
| perl ./program2 - -           \
| perl ./program3 - prog3.output

etc

    [download]

    Even better would be to add some logic to your scripts so that they defaulted to "-" automatically if filenames were absent, and then you could do ...

      perl ./program1   prog1.input \
| perl ./program2               \
| perl ./program3 > prog3.output

etc

    [download]

        --k.


[reply]
[d/l]
[select]
Re: UNIX shell scripts + pipes
by broquaint (Abbot) on May 27, 2002 at 14:36 UTC
    You could use pipes from the looks of it 
    # $@ being the shell's quoted argument list
perl program1 $@ | perl program2

    [download]
    Also you may want to make your perl scripts executable for clarity's sake.

    Another thing to note is when you're piping into a perl program the output from the source program will be on STDIN e.g 

    # program1.pl
...
print while(<SOMETHING>);

# program2.pl (which is reading program1.pl's output)
...
while(my $line = <STDIN>) {
    do_something($line);
}

    [download]
    There's also the handy -n switch and many others of whose information you can find in the perlrun manpage.
    HTH

    _________
    broquaint

[reply]
[d/l]
[select]
Re: UNIX shell scripts + pipes
by penguinfuz (Pilgrim) on May 27, 2002 at 15:50 UTC
    ...i am trying to write a shell script that will run 5 of my programs in sequential order.
    Are these programs shell scripts, perl scripts, or what?

    ...however i don't know how to do it so that the input files can be different
    I do not quite understand this one.

    ...i also want to pipe the out put file so that it can be the input file of the next program.
    If the 5 programs are perl scripts why not condence them into 5 subroutines included in the same script? ;) If the programs are shell scripts just feed their output into an array and process by passing each value to another subroutine.

    Assuming you are taking output from running the other scripts, here is an example of my understanding.

    3 scripts: start.pl, hello.pl, and chopup.pl

    start.pl
    #! /usr/bin/perl -w
use strict;
my @program1 = `perl ./hello.pl`;
foreach (@program1) {
  # pass to program2
  &program2($_);
}

sub program2() {
  # value from program1
  my $value = shift;
  my @newvalue = `echo $value | perl ./chopup.pl`;
  foreach (@newvalue) {
     print $_;
   }
}

    [download]
    hello.pl
    #! /usr/bin/perl -w
use strict;
my $message = "Hello Lucy!";
print "$message";

    [download]
    chopup.pl
    #! /usr/bin/perl -w
use strict;
chomp (my $value = <STDIN>);

my @value = split "",$value;
my $count = @value;

my $i;
for ($i=0;$i<$count;$i++) {
  print "\@value[$i] = $value[$i]\n";
}

    [download]
    Run ./start.pl and all files will be processed. I hope this helps, or maybe I should say I hope my understanding is correct! ;
[reply]
[d/l]
[select]
Re: UNIX shell scripts + pipes
by kodo (Hermit) on May 27, 2002 at 15:00 UTC
    I'm not sure if I understood you right, but I guess you want something like this:

    #!/bin/ksh
FILE=$1

COUNTER=1
perl ./program0 "${FILE}" "${FILE}.${COUNTER}"

while [ $COUNTER -lt "5" ] ; do
   perl ./program${COUNTER} ${FILE}.${COUNTER} ${FILE}.$(expr ${COUNTE
+R} + 1)
   COUNTER=$(expr $COUNTER + 1)
done

    [download]


    You could start it with script.sh<FILE> and it would call program1-5 and give you 5 output files, and the last one would have gone through all five...but shell-scripts are evil :)

    giant
[reply]
[d/l]
[select]
Re: UNIX shell scripts + pipes
by thor (Priest) on May 27, 2002 at 23:15 UTC
    In Bourne Shell (only 3 programs, but that's cuz I'm lazy): 
    #!/bin/sh -x
FILE1=$1
FILE2=$2
FILE3=$3
FILE4=$4

prog1.pl $FILE1 $FILE2
STATUS=$?
if [ $STATUS != 0 =; then
 set +x
 echo "Belch in prog1.pl: $STATUS"
 exit $STATUS
fi

prog2.pl $FILE2 $FILE3
STATUS=$?
if [ $STATUS != 0 =; then
 set +x
 echo "Belch in prog2.pl: $STATUS"
 exit $STATUS
fi

prog3.pl $FILE3 $FILE4
STATUS=$?
if [ $STATUS != 0 =; then
 set +x
 echo "Belch in prog3.pl: $STATUS"
 exit $STATUS
fi

    [download]
    Or, if the intermediate file names don't matter, and the intermediate files aren't too big: 
    #!/bin/sh
FILE1=$1
FILE2=/tmp/file.$$.1
FILE3=/tmp/file.$$.2
FILE4=$2
SAFE=/path/to/save_file.$$

prog1.pl $FILE1 $FILE2
STATUS=$?
if [ $STATUS != 0 =; then
 set +x
 echo "Belch in prog1.pl: $STATUS"
 exit $STATUS
fi

prog2.pl $FILE2 $FILE3
STATUS=$?
if [ $STATUS != 0 =; then
 set +x
 echo "Belch in prog2.pl: $STATUS"
 mv $FILE2 $SAVE
 exit $STATUS
fi

prog3.pl $FILE3 $FILE4
STATUS=$?
if [ $STATUS != 0 =; then
 set +x
 echo "Belch in prog3.pl: $STATUS"
 mv $FILE3 $SAVE
 exit $STATUS
fi

#Clean-up temp files
/bin/rm -f $FILE1 $FILE2
STATUS=$?
if [ $STATUS != 0 =; then
 set +x
 echo "Error removing temporary files"
 echo "$FILE1 and"
 echo "$FILE2."
 echo "Please clean-up by hand"
 exit $STATUS
fi

    [download]
    That answers your original question. That having been said, the answers given above are much better ideas. This way does give you a couple of break points so that if prog4.pl chokes, you still have the output from prog3.pl and a place to start debugging prog4.pl

    Hope this helps, thor

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom