URL-String-concatenation within Perl [applying to a loop]

Perlbeginner1 has asked for the wisdom of the Perl Monks concerning the following question:

Hello dear Perl-monks - good evening!

well i am currently workin on a parser that should fetch pages: i want to do it with Perl since this is the most pretty way to do it!
well - i have to build a loop since i need to fetch many pages from one server:
siteone_dot_com?show_subsite=9009
siteone_dot_com?show_subsite=9742
siteone_dot_com?show_subsite=9871

Should i do it like this?

for my $i (0..10000) {
  $ua->get('siteone_dot_com?show_subsite', id => , extern_uid => $i);
  # process reply
}
[download]

Well i am not sure bout the string concatenation. How to do it - Note i want to fetch several hundred pages:

see the some details for this target-server sites - /(i have to create a loop over several hundred sites) siteone_dot_com?show_subsite=9009
siteone_dot_com?show_subsite=9742
siteone_dot_com?show_subsite=9871

...and so on and so fort... Question: How to appy this loop into the code!?

  use strict; 
  use warnings; 

    use WWW::Mechanize;
    use File::Basename; 

    my $m = WWW::Mechanize->new;

    $m->get("the domain name ");
[download]

love to hear from you!
regards

Comment on URL-String-concatenation within Perl [applying to a loop] Select or Download Code

Replies are listed 'Best First'.
Re: URL-String-concatenation within Perl [applying to a loop] by 7stud (Deacon) on Nov 27, 2010 at 19:32 UTC
WWW::Mechanize defines get() like this: `$mech->get($uri)` So you could simply do this: `my $mech = ....; for my $page_number (1 .. 1_000_000) { my $uri = "siteone_dot_com?show_subsite=$page_number"; my $response = $mech->get($uri); }` [download] But it also appears you might be able to do something like this: `my $mech = ....; my $url = "siteone_dot_com"; my $param = 'show_subsite'; for my $value (1 .. 1_000_000) { my $response = $mech->get($url, $param => $value); }` [download] Experiment and see if they both work.	[reply] [d/l] [select]
Re^2: URL-String-concatenation within Perl [applying to a loop] by Perlbeginner1 (Scribe) on Nov 27, 2010 at 20:09 UTC
hello 7stud, many many thanks - i will experiment and have a closer look at both versions! i come back and report all my findings! greetings pb1	[reply]
Re: URL-String-concatenation within Perl [applying to a loop] by morgon (Priest) on Nov 28, 2010 at 05:33 UTC
I am really sorry to see that you don't seem to make a lot of progress with your project (you've posted quite a bit about it previously) - at least that's what it looks to me... While what you are trying to achieve is absolutely doable (and Perl is an excellent choice for it) I think you should admit to yourself that you lack a bit of background to really get going... Please don't take it personally but you seem to just randomly fiddle with pieces of code that you get from man-pages or god-knows-where without really understanding what you are doing... While this can work sometimes it is just not a very clever strategy for the long run. So I would advise you to work through a good introduction (Learning Perl maybe?) before you start hacking. That won't take you long and should improve your productivity significantly. I really hope you teach your pupils better learning techniques than the ones you exhibit...	[reply]
Re: URL-String-concatenation within Perl [applying to a loop] by aquarium (Curate) on Nov 28, 2010 at 22:55 UTC
your brute force loop to construct urls will exhibit thousands of non-existing urls. this will hammer the server for no good reason, and make your code run for hours instead of minutes. the site administrator will notice all the failed attempts (bad generated urls) and it might develop into your ISP contacting you. In any case, it is very bad net etiquette, and a waste of resources. Plus you insist on perlmonks helping you in this effort. have you yet bothered to ask the site admin if the proper url list is available via api or another page?..because this sort of contact is preferrable to getting noticed the other way. the hardest line to type correctly is: stty erase ^H	[reply]
Re: URL-String-concatenation within Perl [applying to a loop] by Anonymous Monk on Nov 27, 2010 at 18:11 UTC
String concatenation? That's easy: `'string literal' . $foo` `qq{string literal$foo}` `join('', 'string literal', $foo)` Now, why don't you try to make that first snippet of code actually compile?	[reply] [d/l] [select]
Re^2: URL-String-concatenation within Perl [applying to a loop] by Perlbeginner1 (Scribe) on Nov 27, 2010 at 19:04 UTC
hi there anon monk thx for the reply. Heard of the two ways - the dot-operator and the join-operator. Well - many many thanks - i will try this out later the weekend... Many many thanks pb 1 ps: i do it with this: `for my $i (0..10000) { $ua->get('siteone_dot_com?show_subsite', id => , extern_uid => $i); # process reply }` [download] btw: see some details for this target-server: http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=9009 http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=9742 http://dms-schule.bildung.hessen.de/suchen/suche_schul_db.html?show_school=9871 i want to loop over the above mentioned results: so i apply the string cocatenation to this examples...	[reply] [d/l]