Here be my code:
_________________________
foreach my $sid (@sids) {
  $sth = Database::query("select SUrl from Sites where SId = '$sid'", 
+$dbh);
  my ( $site ) = $sth->fetchrow_array();
  push @reqs,  HTTP::Request->new(GET => $site);
  push @url_list, $site;
  $sth->finish;
}
my ($req,$res);
#Creating a new LWP UserAgent
my $pua = LWP::Parallel::UserAgent->new();
# sets the maximum number of requests, and the maximum number of host
# allowed at the same time to 15
# set the UserAgent to connect to requests in order.
$pua->max_hosts(15);
$pua->max_req(15);
$pua->in_order(1);
# Registers the requests so the useragent knows which websites to go t
+o.
foreach my $req (@reqs) {
  $pua->register ($req);
}
# executes requests
my $entries = $pua->wait();
my $counter = 0;
foreach (keys %$entries) {
     my $res = $entries->{$_}->response;
     my $content = $res->content;
     my $url = @url_list[$counter];
     # ... update the copy of the site content in Sites.SLatestVersion
     my $sth = $dbh->prepare("update Sites set SLatestVersion = ?,SLas
+tUpdated = NOW() where SUrl = ?", $dbh) or die "Couldn't prepare stat
+ement!" . $dbh->
     $sth->execute( $content, $url );
     $sth->finish;
     # retrieve site ID for that specific webpage
     $sth = Database::query("select SId from Sites where SUrl = '$url'
+", $dbh);
     my $sid = $sth->fetchrow_array;
     $sth->finish;
     # set DDiffExists to zero for all entries in the Docs table for t
+his SId
     $sth = Database::query("update Docs set DDiffExists = '0' where D
+TagSId = '$sid'", $dbh);
     $counter++;
}

[download]
_____________________________
The problem is that the parallel lwp remaps the url. e.g. at the start the url is http://www.toonami.com but by the end the url has become http://www.cartoonnetwork.com/toonami/. How do I find out the original url? (or some other solution that gets around the problem)

In reply to Parallel LWP by jplan34

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.