in reply to Clusters, Distributed Computing, and Perl

I am sorry to be a wet blanket, but why are you doing this work? Projects done in a vacuum because its author thinks it would be cool to work on generally don't go anywhere useful. Particularly when there are multiple other projects that do the same thing already.

If you have a real computational problem to solve, then install Mosix, write your logic using fork when you can, and call it a day. If you want failover, write it as a web application, avoid silly things like shared memory, and put a load balancer in front of it. If your problem is embarrassingly parallel, but you need different machines to co-operate, don't bother with an official cluster. Instead communicate through a database, and keep a current status table of jobs that need to be done, have been started, etc.

Each of these has been called a cluster, they each solve different problems, and all are usable right now in Perl.

  • Comment on Re: Clusters, Distributed Computing, and Perl

Replies are listed 'Best First'.
Re: Re: Clusters, Distributed Computing, and Perl
by hossman (Prior) on Oct 13, 2002 at 19:53 UTC
    Projects done in a vacuum because its author thinks it would be cool to work on generally don't go anywhere useful. Particularly when there are multiple other projects that do the same thing already.

    Just because an endeavour doesn't benefit "Mankind" doesn't mean it doesn't benefit the Man undertaking the endeavour -- And an experiment conducted over and over by thousands of people in the past is still worth conducting one more time by someone who is not one of those thousands of people.

    It's called education.

[reply]
      The person doing the experiment again usually gets that benefit more efficiently if they start by learning what the people who already did the experiment thought they learned.

      Which is why the education system doesn't first teach you about gravity by giving you two cannonballs and pointing you at a likely tower.

      This is not to say that there aren't good reasons to have people do things again. You aquire useful skills. The knowledge sticks better. It may be entertaining. It gives an appreciation for what the research entails. And the first few times it is necessary to show that the experiment is reproducible and doesn't depend on hidden factors that the researcher didn't see.

      In this case the benefits are rather one-sided. Even if he eventually writes his own clustering sytem, he will learn more if he first studies what people already know about the topic.

[reply]
        Enigmae has already looked for prior work; he said that at the beginning of his post. It seems to me that he's only looking to start from scratch if what he's looking for isn't available. In the event that he must roll his own, his goal doesn't seem to be simply learning:

        What I had in mind was to use perl to create a powerful application framework that can provide a mechanism to easily multi-task and solve problems using several machines and be easy to intergrate in existing and future projects.

        He has a very specific need to fill. Systems (like Mosix) do exist which could solve the problem, but it doesn't simply follow that others shouldn't be created. Since he says he needs to combine with "existing projects," the existing solutions may not work for him. His existing logic may not map well to fork(). He may need to use it in conjunction with Solaris, WinNT or Plan9.

        Anyway, if somebody wants to make a pure perl solution for something, more power to 'em. I could use it for what I'm doing right now, currently non-parallel, across the Linux/Solaris divide.

        --

        Love justice; desire mercy.
[reply]

In Section Seekers of Perl Wisdom