use LWP::UserAgent repeatedly

Perlbeginner1 has asked for the wisdom of the Perl Monks concerning the following question:

Hello dear monks good morning

I am trying to use LWP::UserAgent on the same URLs see below with different query arguments, and i am wondering if LWP::UserAgent provides a way for us to loop through the query arguments?
I am not sure that LWP::UserAgent has a method for us to do that.
I tried to figure it out. And i digged deeper in the Manpages and Howtos: we can have a loop constructing the URLs and use LWP::UserAgent repeatedly:


for my $id (0 .. 100000)
{
    $ua->get($url."?id=21&extern_eid=".(09-$id))
    //rest of the code
}
[download]

or - much better - i do that:


for my $i (0..10000) {
  $ua->get('http://www-db.sn.schule.de/index.php', id => 21, extern_ui
+d => $i);
  # process reply
}
[download]

Well, alternatively we can add a request_prepare handler that computes and add the query arguments before we send out the request.

Do you think that this fits the needs? look forward to your review of this issue!
greetings
beginner

What is aimed: Here on this following site we find a list of many schools: see the page with the subsequent results - approx more than 1000 sites

http://www-db.sn.schule.de/index.php?id=25

i want to fetch all the sites that are listet on this page - and therefore i want to use LWP::UserAgent; - and subesquently i want to parse them.

the sites can be reached directly - by constructing in other words the subsites of the overview can be reached via direct links... see the following.

http://www-db.sn.schule.de/index.php?id=21&extern_eid=1543
http://www-db.sn.schule.de/index.php?id=21&extern_eid=709
http://www-db.sn.schule.de/index.php?id=21&extern_eid=789
http://www-db.sn.schule.de/index.php?id=21&extern_eid=1297

well long speech short sense: - i try this out - and let you know all the findings

Comment on use LWP::UserAgent repeatedly - with a loop Select or Download Code

Replies are listed 'Best First'.
Re: Loop constructing the URLs to use LWP::UserAgent repeatedly up to 10 thousand times by ww (Archbishop) on Oct 23, 2010 at 04:42 UTC
Perlbeginner1: You really need to start reading the documentation for the modules you try to use and... start trying to answer your own questions. This one is the 4th or 5th about your project in about as many days... and, like the others, could be answered on your own, with just a bit of initiative.	[reply]
Re^2: Loop constructing the URLs to use LWP::UserAgent repeatedly up to 10 thousand times by Perlbeginner1 (Scribe) on Oct 23, 2010 at 04:47 UTC
hi there many thanks ww for answering. well thx for the hint. i read the manpage and i am almost sure that i can do this...The loop-construction is sure doable! i try out and report all my findings! regards beginner!	[reply]
Re: use LWP::UserAgent repeatedly - with a loop by marto (Cardinal) on Oct 23, 2010 at 10:47 UTC
You've posted similar site scraping/HTML parsing questions yet you don't seem to be taking the advice you've already been given. getting started with LWP and HTML::TokeParser for example. Also, please check the formatting of your posts and site sites formatting advices. Your "see below" link doesn't work	[reply]
Re: use LWP::UserAgent repeatedly - with a loop by ikegami (Patriarch) on Oct 24, 2010 at 08:44 UTC
`use URI qw( ); sub build_url { my $url = URI->new(shift(@_)); $uri->query_form(@_); return $uri; } $ua->get(build_url('http://www-db.sn.schule.de/index.php', id => 21, e +xtern_uid => $i));` [download] URI	[reply] [d/l]