in reply to The State of Parallel Computing in perl 2007?

I don't know if you would consider it parallel programming or not, but I use memcached to get more than one computer into the act. There are several perl modules that use it.

Unlike many things in parallel programming, memcached is easy.

POE::Wheel::Run will allow you to use multiple processes, which should provide parallelism on a multiprocessor machine. To use it in windows, I used Cygwin since POE::Wheel::Run required the more full-featured fork/exec.

Another easy way to do parallel computing is to use a web server. One program can make requests to multiple servers, or a single server with multiple CPUs, and get them all working on parts of a problem. You can use POE to control the flow of making multiple web requests and synching back up when they return.

It should work perfectly the first time! - toma
  • Comment on Re: The State of Parallel Computing in perl 2007?

Replies are listed 'Best First'.
Re^2: The State of Parallel Computing in perl 2007?
by jettero (Monsignor) on Jan 22, 2007 at 11:58 UTC

    I saw in the POE docs that there were references to needing a serializer. I didn't get far enough with it to see that you can fork. Does it support load balancing and things? I've got the POE::WHeel::Run docs up presently, and I'm seeing that it forks a child process, but I thought it was for things like spawning 'cat' or 'ls' or whatever...

    That probably isn't what I have in mind, but I do still wonder if POE has some kind of built in shared memory multi-processor and/or multi-computer features. It seems like it should.

    -Paul

[reply]
      I use POE::Wheel::Run to spawn four Perl programs.
      1. A web server built from HTTP::Daemon. This provides a browser-based GUI. This web server also spawns programs that can get content from other web servers.
      2. A live link to a large CAD program.
      3. A live link to a large circuit simulator.
      4. A terminal that provides user messages and a command line, for development and for cases where the GUI doesn't have deep enough functionality.
      I hadn't thought about this program as parallel processing until I saw your question. I have only recently begun running the application on multi-cpu machines.

      The program uses message passing through several mechanisms:

      • STDIN, STDOUT, and STDERR of child processes.
      • Dropping files. The CAD package uses this for input.
      • Web calls. I recently switched from LWP to curl because of deployment difficulties. I had trouble getting my installer to automate the configuration of LWP.
      • Environment variables are used to send parameters into child programs. I had trouble with platform differences in the handling of command-line arguments. This is possibly due to differences in quoting and escaping.
      It should work perfectly the first time! - toma
[reply]
Re^2: The State of Parallel Computing in perl 2007?
by perrin (Chancellor) on Jan 23, 2007 at 18:55 UTC
    You use memcached as a database? Not a good idea. Use a database for that. Memcached doesn't consider it a problem to drop your data silently if it runs out of free RAM. That's typical for a cache.
[reply]
      No, I don't think I mentioned using memcached as a database. Memcached allows me to use multiple machines to cache data in a transparent manner. It allows me to use more, cheaper machines rather than one big expensive machine. I have heard people describe this tactic as "build out, not up."

      A common use case is to cache the results of an SQL query. I use the query as the cache key and the value is the result set from the query. I check the cache to see if the dataset is there. If it is, I get it from the cache. If it isn't, I run the SQL and put the results in the cache. This provides me with a huge speedup.

      If the data in the database gets updated, I flush the cache and start again. This is not a problem for parts of my application, so those are the parts where I use memcached. Instead of storing lot of data in a perl data structure in mod_perl program and counting on the copy-on-write mechanism to save RAM, I use memcached.

      It should work perfectly the first time! - toma
[reply]
        Sorry, when you said "I use memcached to get more than one computer into the act" I took that to mean you were using it as your way of sharing data between machines, rather than using a database. The usage you described here makes perfect sense.
[reply]

      memcached is just about the coolest thing ever btw. It's on the top of my list of things to learn next. UPDATE: oops, wrong parent. Eh.

      -Paul

[reply]

In Section Seekers of Perl Wisdom