Beefy Boxes and Bandwidth Generously Provided by pair Networks
Come for the quick hacks, stay for the epiphanies.
 
PerlMonks  

Re: Parallel Processing, Queueing and Scheduling

by AssFace (Pilgrim)
on Mar 12, 2003 at 16:15 UTC ( [id://242406]=note: print w/replies, xml ) Need Help??


in reply to Parallel Processing, Queueing and Scheduling

There are three ways that come to mind. COW (cluster of workstation - like the grid computing that one of the responses on here mentions), Beowulf type clustering, and then MOSIX/OpenMOSIX type clustering (which is sort of a mix of the two).

As far as I know, none of those allow for cross platform.

From what I know of parallel work - which from what I have done is graphics and financial data, but I wouldn't really say I'm an expert - you want to have your processing done in C.
Which it sounds like you are doing - it sounds as if you want something to handle sending off data to each node on the cluster.

People already talked of beowulf (the pvm and/or mpi stuff is used on that implementation), and they have talked of grids - but I didn't see Mosix on here.

When I was starting up learning all of this I was interested in doing a Beowulf cluster because... well, they sound cool. But then I was starting to see a trend where for the things that I wanted to do, it was actually adding a level of complexity that wasn't needed.
So I dumped on the Beowulf idea and went with OpenMosix.

You can obviously read up more on your own of course, but the general idea is that you have N nodes in a cluster - all running Linux, with the kernel mods that OpenMosix requires.
Then from there, you have a few options - the one I am more familiar with is having one head node that keeps track of what the others are up to and what loads they are under. You then can put your perl script on the head node and run it on there, and every time that you want to feed off the processor intensive part, then you fork off your call to the C program (using the Perl ForkManager module) and OpenMosix will pass that off to the node that is the least busy at that moment.
There is also a varient that has more of the grid idea where everyone's computers run it and they are basically workstations, but can also compute stuff in the background if they are freed up.

The basic concept of it is that they have a shared network filesystem (PFS - different than NFS, but similar too <g>). They don't have the shared memory of an SMP system, so the bottleneck tends to reside in the network speed.


A general example for my work would be that I have my cluster, and I ssh into the head node and run the perl script, and then that spawns off the C program, passing it parameters so that it can do its thing, and then it saves out to the disk - then once the bulk processing is done, in my case the perl collects the data and makes one collective document.
In your case it would vary depending on what rendering you are doing (if you are doing it where each node processes a pixel, then it is *very* different than if you are doing it so that each node is assigned to render out the movie frames N through N+10 and then later put those frames together into a movie).

I'm not sure how well I explained all of that - but the shortest answer is to read up on OpenMosix and it is likely that someone out there is doing something very similar to what you are wanting to do.
  • Comment on Re: Parallel Processing, Queueing and Scheduling

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://242406]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others browsing the Monastery: (3)
As of 2024-04-26 08:27 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found