Running a shell script from Perl, and making that shell script accept input from perl script.

why_bird has asked for the wisdom of the Perl Monks concerning the following question:

Morning monks.

This is a question about running a shell script from a Perl script, and then making the shell script wait and accept input from the perl script. I can only submit jobs to our queueing system via a shell script, but I want to do other stuff (involving e.g. Tie::File) that would be a real pain to re-write in a shell script. Recalling the shell script every time a job is generated and passing it the data from the Perl script is not an option, neither is waiting until all the jobs are generated and passing them en masse. It needs to start up, and then accept input as and when it's generated from the Perl script. The shell script basically does (pseudo code):

Configure the grid engine to receive a batch of jobs
foreach job @jobs{
Submit job
}
[download]

The way I'm tackling this at the moment is to replace the foreach job loop with a loop like this:

#Job.sh

read arg
while [ arg != TERMINATE ]
do
(submit job arg)
read arg
done
[download]

The real crux of the question is this: How do I run the above shell script from a Perl script, then make it wait for input from STDIN whilst allowing the perl script to keep running? Currently, I've got something like this, which obviously doesn't work..:

#SubmitJob.pl
use strict;
use warnings;

my @jobs;

# do some stuff here...
...
# run the shell script

`./Job.sh`;

# generate the jobs..
@jobs=...

# print the jobs to STDOUT so that they can be picked up by the shell 
+script:
foreach my $job (@jobs){
print $job;
}

exit 0;
[download]

Now, I understand why this doesn't work (the Perl script won't keep running if the shell script is started in the foreground, but the shell script won't accept input from STDIN if it's started in the background). But I don't know how to make it do what I want. My only thought was a fork---is this appropriate? Or can someone suggest a whole new approach to doing this?

thanks, why_bird

edit changed title to avoid confusion; readmore'd some text

........
Those are my principles. If you don't like them I have others.
-- Groucho Marx
.......

Comment on Running a shell script from Perl, and making that shell script accept input from perl script. Select or Download Code

Replies are listed 'Best First'.
Re: Running a shell script from Perl, and making that shell script accept input from perl script. Fork? by Corion (Patriarch) on Apr 08, 2008 at 07:50 UTC
In the most general case, you want some IPC. In your specific case, the pipe-open is likely the easiest way to submit jobs, as you're not interested in the output of your job launcher: `my $queue = './Job.sh'; open my $submit, '\|-', $queue or die "Couldn't launch queue submission '$queue': $!/$?"; ... for my $job (@jobs) { print {$submit} "$job\n"; };` [download]	[reply] [d/l]
Re^2: Running a shell script from Perl, and making that shell script accept input from perl script. Fork? by why_bird (Pilgrim) on Apr 08, 2008 at 08:28 UTC
Thanks---I didn't realise that's what pipes were for. It's working very well now. Cheers why_bird ........ Those are my principles. If you don't like them I have others. -- Groucho Marx .......	[reply]
Re: Running a shell script from Perl, and making that shell script accept input from perl script. by jsegal (Friar) on Apr 08, 2008 at 23:25 UTC
Which queuing system are you using? I have used SGE, which requires a shell script for submission, but it is easy enough to write a single shell script wrapper, effectively: `--wrap.sh-- #!/bin/sh "$@"` [download] --------- If you look at the shell docs, you will see that the $@ expands to all the arguments, and putting it in quotes ensures that any spaces in any of the argument values will still be quoted (i.e. they won't cause arguments to be split on spaces). Then one can submit thusly: `qsub -o <log> -q <queue> [other qsub args] wrap.sh perl scriptname scr +iptargs` [download] If you have a submitting job and the sub jobs that need to communicate, you can still use pipes if that makes sense. And obviously if you have the proper shebang line in your perl script you don't need to call perl explicitly. Perhaps this trick will help you think of other ways to solve your issue. --JAS	[reply] [d/l] [select]


Keep It Simple, Stupid
	PerlMonks