in reply to Re^12: Problem in Inter Process Communication
in thread Problem in Inter Process Communication

can't i use fork to fork multiple processes rather than threads?

Yes you can. But again, what benefits are you seeking to gain?

You could also use a single threaded, single process and non-blocking IO. I wouldn't recommend it especially as you have to discrete parts to your processing, with significantly different processing time requirements: 1 IO bound, 1 CPU bound; but you could do it that way.

If you had the time and interest, coding the same problem all 3 ways would be an interesting comparative exercise, but people rarely have the time or budget for such explorations.

The nice thing about the single-queue, worker-pool threaded solution is that it is really easy to get going--you've almost written it three times during these discussions--and once it working, it scales very easily and well.

It is a great way to prototype something that works and then use that to explorer where the limits and bottle necks are. Once you know whether, at the limits of your hardware, you are IO-bound, cpu-bound, or memory-bound, you can use that information to consider if one of the other solutions would allow you to transcend that boundary.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."
  • Comment on Re^13: Problem in Inter Process Communication

Replies are listed 'Best First'.
Re^14: Problem in Inter Process Communication
by libvenus (Sexton) on Aug 22, 2008 at 12:08 UTC

    hmm, well that makes sense.. Why i asked this question is because i need to give benefits of my approach( threaded) to the earlier one(forking) to the architects.I plan to propose the simple thread::queue model.I need to tell them y is it scalable, maintainable and better..

[reply]
      need to tell them y is it scalable, maintainable and better...

      Unfortunately, there's no way that you'll get any serious developer to proffer an opinion as to which mechanism would unltimately be the "best" strategy on the basis of the minimal amout of imformation you've provided.

      • What type of application servers are involved?
      • What type of queries are you "firing off"?
      • What kind of network bandwidth will the server have?
      • What kind of network latency is involved?
      • What type of "comparison"s are you doing? (Simple string compare or something more complicated.)
      • Why are you redeveloping? </il>
      • What is wrong with the existing forking solution?
      • What language was employed in thae previous solution?
      • Will the 12-cpu box be dedicated to just this task?
      • What are the project requirements?
      • Which of those is the existing code not meeting? And by how much is it missing it or them?

      And that's just off the top of my head. With a proper analysis, that list would grow 10-fold. And even if accurate answers where available (and possible to determine), to all of those questions, it still leaves all kinds of questions about the development processes. And the experience levels of the developers involved. You'd still be hard pushed to get anyone to put their hand on their heart (never mind their wallet), and give you a definitive statement as to which approach would be the "best". In any of the possible meanings of best.

      I'd tackle it using threads as my first pass, because that's where my experience and expertise lies. And because assuming that the querying code and comparison code is available, I could have a testable protype up and running in a matter of hours. Once I had that prototype, I could generate some data upon which to evaluate the effectiveness of that prototype and the possibilities for meeting the project requirements.

      I hope that provides you with some help in your decision making process.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

        Hi BrowserUk,

        Please find the answers inline

        What type of application servers are involved?

        1. These servers are an i/f to the db so instead of firing query to a specifc table all we have

        What type of queries are you "firing off"?

        2. The queries are simple select queries though can have 100 colummns and can retrun 10K rows as well.

        What kind of network bandwidth will the server have?

        3. Not Sure

        What kind of network latency is involved?

        4. Not Sure

        What type of "comparison"s are you doing? (Simple string compare or something more complicated.)

        5. The o/p returned from prod and test are first string compared . if there is some error then we go into lower levels of comparison.The app server have a charecteristics of messing up with the order of the results returned . so they are sorted first then row by row comparison takes place.There are various clauses that one needs to take care one is exclude certain columns from comparison, compare only certain keys ( these are just a few) in all the comparison is not simple

        Why are you redeveloping?

        6.1 The QA team would not have to rush to the devlopers each time there is some problem or some enhacement.They know perl better.

        6.2 Perl is widely known two be very good is string comparison.

        6.3 I have been told by my Lead)(i have to make it and make it good else i would be kicked out :-))

        What is wrong with the existing forking solution?

        7.1 It is not scalable/extendable

        7.2 Comparison logic takes a lot of time.

        What language was employed in thae previous solution?

        8. C++

        Will the 12-cpu box be dedicated to just this task?

        9. That i what i have been told but it can go down to 4 cpu also

        What are the project requirements?

        10. Rewrite/Redesign the C++ code in Perl 5 so that it takes lesser time and is maintaineable.

        Which of those is the existing code not meeting? And by how much is it missing it or them?

        11. The Suite takes 3 weeks to run i mean the entire cycle i have to bring it down

[reply]

In Section Seekers of Perl Wisdom