Heya fellow Monks,

This is an issue I've been dealing with for some time now. I need to make multiple HTTP requests from within a mod_perl2 handler. Each request takes a variable amount of time (between 1 and 2 seconds), and all requests must be completed or aborted on a fixed time (call it 3 seconds).

I'm currently using LWP::Parallel::UserAgent to accomplish the concurrent requests, wrapped in an eval block and using SIGALRM/alarm to enforce the fixed timeout (setting the timeout of LWP::Parallel::UserAgent, as with LWP::UserAgent, is the timeout between inactivity, so a slow network can extend the request almost indefinitely). The SIGALRM/alarm seems to work fairly well, but it doesn't seem like the ideal solution.

I'm stuck using Apache2's prefork worker model (the threaded model looks like it'll perform better with my application) but some of the code somewhere isn't threadsafe. I can't say for certain that it's LWP::Parallel::UserAgent, but that's one suspicion.

I don't think I'm doing anything new here, so I'm hoping someone else has been through this and found a good solution. I've done extensive googling and digging through CPAN with no luck. I've considered writing a new, thread-safe alternative to LWP::Parallel::UserAgent. I've also thought about writing a seperate daemon to act as a proxy of sorts to the mod_perl2 handler, which would do all the downloads multi-threaded. I can't say I'm excited about the time involved in either prospect, but I'll do what it takes to get this working smooth.

In reply to Parallel HTTP requests under mod_perl2 by StoneTable

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.