Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

comment on

( [id://3333]=superdoc: print w/replies, xml ) Need Help??
There are three ways that come to mind. COW (cluster of workstation - like the grid computing that one of the responses on here mentions), Beowulf type clustering, and then MOSIX/OpenMOSIX type clustering (which is sort of a mix of the two).

As far as I know, none of those allow for cross platform.

From what I know of parallel work - which from what I have done is graphics and financial data, but I wouldn't really say I'm an expert - you want to have your processing done in C.
Which it sounds like you are doing - it sounds as if you want something to handle sending off data to each node on the cluster.

People already talked of beowulf (the pvm and/or mpi stuff is used on that implementation), and they have talked of grids - but I didn't see Mosix on here.

When I was starting up learning all of this I was interested in doing a Beowulf cluster because... well, they sound cool. But then I was starting to see a trend where for the things that I wanted to do, it was actually adding a level of complexity that wasn't needed.
So I dumped on the Beowulf idea and went with OpenMosix.

You can obviously read up more on your own of course, but the general idea is that you have N nodes in a cluster - all running Linux, with the kernel mods that OpenMosix requires.
Then from there, you have a few options - the one I am more familiar with is having one head node that keeps track of what the others are up to and what loads they are under. You then can put your perl script on the head node and run it on there, and every time that you want to feed off the processor intensive part, then you fork off your call to the C program (using the Perl ForkManager module) and OpenMosix will pass that off to the node that is the least busy at that moment.
There is also a varient that has more of the grid idea where everyone's computers run it and they are basically workstations, but can also compute stuff in the background if they are freed up.

The basic concept of it is that they have a shared network filesystem (PFS - different than NFS, but similar too <g>). They don't have the shared memory of an SMP system, so the bottleneck tends to reside in the network speed.


A general example for my work would be that I have my cluster, and I ssh into the head node and run the perl script, and then that spawns off the C program, passing it parameters so that it can do its thing, and then it saves out to the disk - then once the bulk processing is done, in my case the perl collects the data and makes one collective document.
In your case it would vary depending on what rendering you are doing (if you are doing it where each node processes a pixel, then it is *very* different than if you are doing it so that each node is assigned to render out the movie frames N through N+10 and then later put those frames together into a movie).

I'm not sure how well I explained all of that - but the shortest answer is to read up on OpenMosix and it is likely that someone out there is doing something very similar to what you are wanting to do.

In reply to Re: Parallel Processing, Queueing and Scheduling by AssFace
in thread Parallel Processing, Queueing and Scheduling by submersible_toaster

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Are you posting in the right place? Check out Where do I post X? to know for sure.
  • Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
    <code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
  • Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
  • Want more info? How to link or How to display code and escape characters are good places to start.
Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chilling in the Monastery: (4)
As of 2024-03-29 15:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found