zakzebrowski has asked for the wisdom of the Perl Monks concerning the following question:

Hi All,

Sanity checking. I have a module which I'm working on. It's currently (tenativly) called Distributed::Run. The motivation for this module is the following:

I have this neato-torpedo client program which either I can use to stress test a network or I can use to solve a problem which takes a while to run. Since I know it works correctly on one host, I should be able to distribute the proccesses to many boxes to either facilitate additional stress testing or distribute the load to multiple boxes. The process also (after a certain period of time) always exits. The problem is that the current Parallel:: modules are a little to complex and aren't (from what I can tell) operating system independent, and it was unclear of how to distribute the files in a consistant manner.

So, I'm designing a module that will (hopefully) be easy to use, will work across multiple os'es (ssh required), and is nice.

So the questions I have...


Thanks...



----
Zak
  • Comment on RFC. Distributed::Run module (idea & namespace)

Replies are listed 'Best First'.
Re: RFC. Distributed::Run module (idea & namespace)
by Zaxo (Archbishop) on Sep 01, 2003 at 04:16 UTC

    That sounds like blood 'n guts perl, so I'd really like to know how it works before committing to a namespace. Namespaces are a high level form of scoping, closely associated with OO Perl. They are specifically restricted to one process on one host - if source rewriting of the module is not invoked. Nobody does that with impunity.

    It sounds like you might find support in the ^Net or ^IPC namespaces, or if not, you might contribute to them. Also look into cluster programming.

    After Compline,
    Zaxo

[reply]

      Hmm... Sweet, there is a Cluster:: name space on search.cpan.org :) I might rename the module to Cluster::Simple or Cluster::Run. The basic way the module & accompying scripts work is (Aprox, still changing slightly):

      • (DONE)Create config ini files
        • host.ini # what hosts to connect to
        • script.ini # what to run on the various hosts and other configuration settings
        • manifest.ini # a manifest of files, which, is updated before the module is run to ensure that the files on the client side & server side have the same md5 check sum
      • Scp config files from server -> clients
      • launch client.pl process
      • Client decides what files it needs and scp's them over
      • Client runs the main process.
      • If multiple proceses are launched at once, waits for a 'processfinishednnn' to be found in the current directory
      • Client scp the diff of the file found in the manifest file back to the orignalserver.
      • Server waits till there is a count on the number of clientdonexxx files in the ./runname directory before it returns back ok.

      In a crappy custom code, I had a linux & solaris box working together. Trying to figure out how to get cygwin's ssh server to actually work, and then I'll declare victory and move on... :) Note, btw, the main process doesn't have to be perl, it could be your favourite exe... :) Also note, will need to get the module approved by 'legal dept.' before it can be distributed to CPAN. :(



      ----
      Zak
[reply]
Re: RFC. Distributed::Run module (idea & namespace)
by tsee (Curate) on Sep 01, 2003 at 09:47 UTC
    While I generally agree with Zaxo that giving advice regarding namespace choice is questionable until we know more about the module, I think the Parallel:: hierarchy should be what you're aiming for. You should not be using a new top level namespace unless modules@perl.org thinks it's okay. (And if they don't react, keep bothering them until the request tracker transition is complete. Whenever that will be.)
[reply]
      I disagree on the name. Parallel::* to me suggests running several similar processes on the same machine. That is not the goal here. <P<But I agree that the OP should make it run first, and demonstrate it can really work as he describes here, cross-platform and all.
[reply]
        I can see your point. Perhaps Parallel:: isn't good after all. I think this is clearly a case to be judged by the modules list. There are some people on it that know CPAN and its conventions extremely well and far beyond what I could judge.
[reply]
Re: RFC. Distributed::Run module (idea & namespace)
by bm (Hermit) on Sep 01, 2003 at 12:42 UTC
[reply]
      Yeah. It looked a little hard to use. If you have had experience with it, I have a few questions:
      • OS dependencies? Can it run on multiple oses (including windows)?
      • Looks like you might need to have it in mind while you are programming your program, not trying to add parallelism after the fact... is this true?
      • What if your main executable updates reguarly? Will it scp the files around for you to accomodate the change?
      • The user guide & manuall are at a 404 address. Is it actively supported?


      ----
      Zak
[reply]
        I do not have any experience with the Perl interface, but do use pvm, specifically pvmgmake. It cut our build times down by 500%.
        One of its aims is to be hetrogenous, so yes it runs on the major OS's (including windows).
        Your question about the main executable is a good one. I have no idea. One would think so...
        I agree that it looks like you need to design your application with pvm in mind, but this looks to be unavoidable. I cannot conceive of a general way to add parallelism after the fact: how the spawned processes synchronisation be completely general?
        I would be surprised if pvm is not still active (their website is still updated):
        http://www.csm.ornl.gov/pvm/pvm_home.html
        http://www.netlib.org/pvm3/
        Not sure about the Perl interface.
        --
        bm
[reply]

Back to Seekers of Perl Wisdom