Oh Monks,

I come to you yet confused again. My learnings teach me that there are always more than one way to do things in perl; however, my walk through search.cpan.org leads me to the conclude that I have found many almost, but not quite, good and reliable choices :)

I am working on an app that make calls to web service apis on various social web sites. Depending upon the results, this could trigger 0 - n additional calls, which could do the same again. At this time, the call depth does not exceed 3.

I have a prototype working using threads that is marginally acceptable in performance.; however, I know that I am losing performance because of the overhead associated with threads::queue and threads::shared. My current boss worker framework, crafted by myself and evolved over a couple of months of work, depends on these two modules to manage requests pipeline and to temporarily store data containing state information as well as data that will ultimately be written to a database.

I am certain that others before me have developed a most streamlined and elegant boss/worker model solution, but I have not been able to find one that works reliably. Now that I have the logic down, I am rexamining my tooling.

I would appreciate any recomendations that you may have on the following:

  1. Asynchronous Boss/Worker for threads
  2. Asynchronous Boss/Worker for fork
  3. Event Loop approach (I am currently leaning toward AnyEvent)
  4. Fast shared perl data structures (references, hashes and arrays)

Goals: reliability, speed

My current target platform is Linux and perl v5.10.1. I can upgrade perl if needed and alternatively can move this to a Windows machine if that were to be advatangeous.

Thanks in advance for your help!

lbe

Updated to strikeout comment on performance penalty of threads::shared and threads::queue based upon feedback from BrowserUK

In reply to thread/fork boss/worker framework recommendation by learnedbyerror

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.