tpederse has asked for the wisdom of the Perl Monks concerning the following question:

Greetings all,

I've been using OpenMPI and C for a little while now, but really would like to be using Perl with it instead. By way of background, MPI allows you to run on a cluster of computers and send work to multiple processors, which for big Perl jobs could be quite nice - MPI is a standard and has been implemented by various places, OpenMPI is the open source version of it.

There are two Perl modules in CPAN that seem like they want to support MPI, but they don't seem to work (both haven't been updated in years and may be abandoned..)

Parallel-MPI

Parallel-MPI-Simple

Anyone have any success in using OpenMPI and Perl using other modules or your own code? Any tips, thoughts and/or advice?

Cordially, Ted

Replies are listed 'Best First'.
Re: Perl and OpenMPI
by dk (Chaplain) on Apr 27, 2009 at 20:02 UTC
    I'd first talk to maintainers of the above modules, possibly sending a patch (years or no years, the Tao teaches that a good programmer doesn't refuse to maintain his code :)

    If that fails, possibly you might go as far as to take maintainership for one of the modules, but again only if you know how to make them work.

      Thanks DK,

      The authors of these modules seem to be toying with karma a bit, since either their email addresses are no longer current or they aren't responding. :) So, problem remains.

      Lacking any other option I might try and develop something, but I'd like to make sure there isn't something out there that is working (at least somewhat) to build off of. These two modules don't even make it past "make" at this point, so it's hard to tell whether or not they are a good starting point.

      Any other options out there that anyone knows of?

      Cordially,
      Ted
Re: Perl and OpenMPI
by iansimpson (Initiate) on May 06, 2009 at 13:07 UTC
    Hi Ted

    I am looking at doing the same thing. My first thought was to use perlcc to convert to C and then use the OpenMPI C-wrapper (but perlcc is a bit of a no no and dropped in Perl5.10). I am looking into whether PAR might help or pp but need to test this all out first.

    I also tried Parallel-MPI and Parallel-MPI-Simple with no joy.

    I should know more after consulting with some HPC experts here at Edinburgh University next week so will let you know then.

    Ian.
      Hi Ted,

      A slightly unsatisfactory follow up note from me I'm afraid. It seems that over here the experience of using Perl with HTC is largely limited to people batching unitised Perl scripts.

      By that I mean people re-writing code to perform operations on a single problem and then feeding large numbers of those onto the nodes through UNIX batch scripts.

      In some ways I guess this is kind of obvious because it means little re-coding is required as opposed to re-writing a Perl script in a parallelised way. I have to ashamedly confess that I am going to do the former also for that reason. I would like to exploit OpenMPI in the future and will seek to do things that way from scratch when I next need to do something intensive on a HPC system.

      Sorry that's not of more help.

      Ian.
Re: Perl and OpenMPI
by jsquyres (Initiate) on May 26, 2009 at 13:22 UTC
    I am unfamiliar with those modules, but do have a larger question to ask...

    The MPI Forum is currently working on MPI 3.0. As part of that process, there is a half-baked proposal about better supporting higher-level languages such as Perl. To be clear: it is unlikely that the Forum will develop official perl bindings for MPI; rather, it seeks to ensure that those who want to create higher-level language bindings would have the appropriate hooks and tools to do so (from a standards perspective). Such issues became clear when the Boost.MPI class library was developed (and other similar projects, such as MPI.NET).

    So my question to the Perl Monks is: what does MPI need to provide to enable someone to write "good" / Perl-ish / Perl-native / Perl-natural MPI functionality? (note that such functionality does *not* need to be a 1:1 mapping to the underlying C MPI bindings)

    A sub question: is there something that Open MPI can provide to better support higher-level language/Perl bindings? (I'm one of the core developers of Open MPI)

    The MPI-3.0 question will likely take a long time to answer (it will be quite a while before MPI-3.0 is finalized). But perhaps there's something interesting / useful that we can do in Open MPI to help Perl MPI functionality in the not-distant future...?

    Thanks!