I'd certainly try to prototype this first with something like FCGI plus some high performance webserver - probably nginx.

There are so many potential problems already solved, so much know-how already available in the human cloud.

Like concept to scale up with a server farm...

"Standing on shoulders of giants" and so on...

And if this "prototype" doesn't deliver sufficiently, one can still learn from the concepts when implementing the DIY server.

Saying so, I'm still in doubt the OP has really identified the bottlenecks of his existing solution.

Cheers Rolf
(addicted to the Perl Programming Language :)
Wikisyntax for the Monastery

In reply to Re^2: Use of do() to run lots of perl scripts by LanX
in thread Use of do() to run lots of perl scripts by chrestomanci

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.