Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Opening simultaneous scripts

by OzzyOsbourne (Chaplain)
on Aug 21, 2000 at 23:11 UTC ( [id://28888]=perlquestion: print w/replies, xml ) Need Help??

OzzyOsbourne has asked for the wisdom of the Perl Monks concerning the following question:

Is there a way to have perl kick off other PERL scripts simultaneously? I don't want the jobs to run in order, I want them all at once. Why? I'm searching servers for media files, and I'm manually kicking of 30 scripts. I would like one script to kick up all 30 at once. The scripts finish in a couple of hours if I run them at once. If they run in a row, it's 30 hours+.

Replies are listed 'Best First'.
RE: Opening simultaneous scripts
by randomblue (Beadle) on Aug 21, 2000 at 23:45 UTC
    Well, if they're simply unrelated scripts (and if you're running Linux or something similar), you could just write a shell script which runs them in the background..
    #!/bin/sh script1 & script2 & . . script30 &

    If that's all you basically need, then writing a complicated Perl program with forking, signal-handling and all that seems a bit of an overkill. :)

      But, don't you see, that using the *shell*. That's blasphemy. No amount of arduous work could ever convince a monk to use a (*gasp*) shell script! Never! ;)

      redmist
      redmist.dyndns.org
      redmist@altavista.net
Re: Opening simultaneous scripts
by tye (Sage) on Aug 21, 2000 at 23:57 UTC
    system(1,"command");

    This is the undocumented (as far as I could find) way to start commands in the background portably for recent versions of Perl. I'm surprised that it is still undocumented.

            - tye (but my friends call me "Tye")
(Guildenstern) RE: Opening simultaneous scripts
by Guildenstern (Deacon) on Aug 21, 2000 at 23:30 UTC
    take a look at tilly's script for running commands in parallel. i don't know if that's what you need or not, but it might be a push in the right direction

    Guildenstern
    Negaterd character class uber alles!

    Update: tye mentioned that IPC::Open3 actually has been ported to Win32, so that might make it a little easier to use tilly's script in Win2K. :)
RE: Opening simultaneous scripts
by OzzyOsbourne (Chaplain) on Sep 07, 2000 at 19:37 UTC

    I found it hard to understand tilly's script because I am pretty new to Perl (I really did try, though). The way I ended up solving this problem was:

    # Created on 9/6/00 by Jonathan E. Dyer @servers=('server1','server2','server3','server4','server5','server6', +'server7','server8','server9'); use Win32::Process; sub ErrorReport{ print Win32::FormatMessage( Win32::GetLastError() ); } foreach $server (@servers){ Win32::Process::Create($ProcessObj, "c:\\perl\\bin\\perl.exe", "perl.exe c:\\findstuff6.pl -s$server", 0, NORMAL_PRIORITY_CLASS, ".")|| die ErrorReport(); #$ProcessObj->Suspend(); #$ProcessObj->Resume(); #$ProcessObj->Wait(INFINITE); }

    The problem that caused me the biggest headache was the third variable in win32::process. I kept forgetting to put perl.exe before the script, assuming that this was covered by variable 2.

    I hope this helps if anyone runs into the same issue...

    -OzzyOsbourne

      Personally I hate using platform specific modules if I can avoid it. When you start with a nice portable language, it is worth a little effort to not lose portability.

      The key to my approach is the IPC::Open3 module and the wait command. The module exports open3, which lets you launch a process with specified STDIN, STDOUT, and STDERR. The return of that is a process ID. Conversely wait will collect the return of a process you launched (the processes are called "children" and this is called "reaping the process" since it is the last step before the process can finish dying) returning the process ID and setting $? to its return code.

      What I did is launched processes hooked directly to my output but supressing all input (because there is no easy way to sort out you being able to talk to 5 processes at once on one pipe) and kept track of what I launched. When I reaped them if there was a problem I would give a message based on what I had. My logic was complicated by the fact that I might have 200 jobs to launch, but don't want to run more than 5 at a time. However that is the idea.

      See if you can't get that to work. If you do you will both have more portable code and will have learned a lot about process communication..both good things IMO. :-)

Re: Opening simultaneous scripts
by qi3ber (Scribe) on Aug 22, 2000 at 21:31 UTC
    Or you can use fork(). Why use system() when you can fork off your own kids. As such:

    my $kids = 0; for ( 1..30 ) { if ( fork ) { waitpid(-1,0) && $kids++; } else { exec('scriptname') or die "Bad Things: $!; } } while ( $kids < 30 ) { waitpid(-1,0) && $kids++; sleep(1); }


    Update: Yes, it's overkill, but you gotta have fun sometime!


       [ qi3ber ] I used to think that I knew it all, then I started to listen.
      When you do that, you need to make sure you're reaping your task kids. Remember that pipe-opens and fork-opens also generate kids that may not be part of your "30 count".

      A better way is to fork your kids, noting their pid as keys in a hash. When you reap a kid, delete it from the hash. If it wasn't part of the hash, it won't affect the count. Then fork new kids (on the block? {grin}) until your hash has 30 keys again.

      I've done this for at least one of my columns, but after 106 by-lines, I can't remember which one anymore. {grin}

      -- Randal L. Schwartz, Perl hacker

        At a guess, one you can't post on your website yet because it is too recent. Anyways, I used a somewhat similar approach in Run commands in parallel in case what merlyn said is opaque to you.

        That way you won't have to wait to find out how to reap a kid. :-)

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://28888]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others contemplating the Monastery: (4)
As of 2024-04-24 17:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found