This is just a simple outline of a web gizmo that could hit a lot of pages simultaneously. I am sure that the one second reponse time per page is due to web, not the Perl parsing of the data.

I haven't written a multi-process LWP web app as the normal sites that I have LWP clients for would probably be upset if I wacked 'em more than a few times per second and I try to be polite. There are also limits as to how many connections you can have open at once...I've never come close in Windows, so I don't know.

The poster asked for psuedo code, and here is one attempt with no error or time out escapes. Fork() in Windows is weird and is actually a thread instead of a separate process.

----
client (single program in this case): 
  maintains a list of requests that it wants
  answers to.. (replies to those requests are required)..
  Maybe this is just a hash table with URL's?

put first request(s) onto request queue then
talk_2_server;

talk_2_server
{

  while (a request hasn't been sent or some request hasn't
         been answered...)
  {
    if (server has reply to a previous request)
      {
         take it off outstanding queue, and deal with it..
         this action will generate additional new requests
         that go onto queue...maybe requests for 20 sub-pages
         ..be careful .. you might overload you are talking to!
      }
    while (I have a new request in queue) { send it to server};
       # might want to think about a "throttle" if hitting 
       # same website
  }
}  

maybe I'm done or I need to loop and stuff more things onto
request queue and talk_2_server again...

---
server:

  I see a new request, fork child to deal with it.
  (Like maybe get the info from URL X). I then wait 
  for next request.

---
child:
  I've got some answer, so I want to send result to client.
  I cooperate with other children to so that I can send
  an "atomic" reponse on the pipe back to client via some
  kind of locking mechanism. Then my job's done, I die.
  Message format could be as simple as first line in the 
  URL you requested...followed by some html response.
[download]