First Web Crawl Task

bennierounder has asked for the wisdom of the Perl Monks concerning the following question:

Hi guys,

I'm very frustrated with this code

#!/usr/bin/perl -w

# a simple web crawler

use strict;
use LWP::Simple;

my $url = shift || die 'Please provide an initial url after filename!'
+;
my $max = 10;

my $html = get($url);
my @urls;
while ($url =~ s/(https:\/\/\S+)[">]//) {
        push @urls, $1;
        print @urls;
}

mkdir "web" , 0755;
open (URLMAP, ">", "web/url.map" ) || die ("can't open web\/url.map\n"
+);
my $count = 0;

for (my $i=0; $i<$max; $i++) {
        my $source = $urls[int(rand($#urls+1))];
        getstore($source, 'web/$count.html');
        print URLMAP "$count\n$source\n";
        $count++;
}
close URLMAP;
[download]

1,17 Top

I run the script, perl web_crawl.pl https://www.money.co.uk and I get this!

perl web_crawl.pl https://www.google.com
Use of uninitialized value $source in concatenation (.) or string at web_crawl.pl line 27.
Use of uninitialized value $source in concatenation (.) or string at web_crawl.pl line 27.
Use of uninitialized value $source in concatenation (.) or string at web_crawl.pl line 27.
Use of uninitialized value $source in concatenation (.) or string at web_crawl.pl line 27.
Use of uninitialized value $source in concatenation (.) or string at web_crawl.pl line 27.
Use of uninitialized value $source in concatenation (.) or string at web_crawl.pl line 27.
Use of uninitialized value $source in concatenation (.) or string at web_crawl.pl line 27.
Use of uninitialized value $source in concatenation (.) or string at web_crawl.pl line 27.
Use of uninitialized value $source in concatenation (.) or string at web_crawl.pl line 27.
Use of uninitialized value $source in concatenation (.) or string at web_crawl.pl line 27.

I'm trying to eventually get the prices and company names, so for example for this part of the site https://www.money.co.uk/travel-money/japanese-yen-exchange-rate.htm I want to get the prices on offer into an array in order (highest first), maybe keeping a note of the company name so may need a hash or array of hashes.
That's the end goal, but stuck on the first hurdle, which is viewing the sites html in files where i can search the prices, then extract them from the files!!! If you can think of a better way and point me in the right direction on finding the solution, I'm all ears! Thanks in advance!

Please help!

Comment on First Web Crawl Task Download Code

Replies are listed 'Best First'.
Re: First Web Crawl Task by marto (Cardinal) on Sep 21, 2018 at 08:13 UTC
Firstly `$source` doesn't contain what you think it does. Tutorials->Debugging and Optimization->Basic debugging checklist. The responses to help with writing a web crawler with LWP may be of interest to you, you should ensure you're not breaking the terms of use of the sites you scrape.	[reply] [d/l]
Re: First Web Crawl Task by roboticus (Chancellor) on Sep 21, 2018 at 11:59 UTC
bennierounder: Looking at your code, there are a couple problems. The one that stands out the most is that you're getting a "Use of uninitialized value" message from your second print loop. That's happening because you're pulling values out of your @urls array that don't exist. You're seeing this because you're looping from 0 to 9 in your printout loop, no matter how many URLs are in the loop. I'd suggest you instead do something like this: `# Let's do this up to 10 times, but only as long as we have URLs while ($count < 10 and @urls) { # Choose a random URL from the list, and remove it from the list my $url_index = int(@urls * rand); my $source = splice @urls, $url_index, 1; # continue normally getstore($source, "web/$count.html"); print URLMAP "$count\n$source\n"; $count++; }` [download] Fixes include: single quotes in your getstore() call would not replace $count with a value, but instead use the string '$count'. we remove each url from the list after use, so you don't reprocess any individual url. we stop the list when you (a) run out of URLs in the list, or (b) have processed 10 URLs. This ought to remove the unsightly warnings you're getting. The bug that's currently biting you, though, is that you don't have any URLs in your list. As you should see, you're printing each URL when you add them to your list, but your code isn't printing any URLs. That's because you're trying to find URLs in your $url string instead of your $html string. Since there are no URLs in https://google.com that are terminated with a quote or right angle bracket, your list comes up empty. ...roboticus When your only tool is a hammer, all problems look like your thumb.	[reply] [d/l]