in reply to First Web Crawl Task
Looking at your code, there are a couple problems. The one that stands out the most is that you're getting a "Use of uninitialized value" message from your second print loop. That's happening because you're pulling values out of your @urls array that don't exist. You're seeing this because you're looping from 0 to 9 in your printout loop, no matter how many URLs are in the loop. I'd suggest you instead do something like this:
# Let's do this up to 10 times, but only as long as we have URLs while ($count < 10 and @urls) { # Choose a random URL from the list, and remove it from the list my $url_index = int(@urls * rand); my $source = splice @urls, $url_index, 1; # continue normally getstore($source, "web/$count.html"); print URLMAP "$count\n$source\n"; $count++; }
Fixes include:
This ought to remove the unsightly warnings you're getting.
The bug that's currently biting you, though, is that you don't have any URLs in your list. As you should see, you're printing each URL when you add them to your list, but your code isn't printing any URLs. That's because you're trying to find URLs in your $url string instead of your $html string. Since there are no URLs in https://google.com that are terminated with a quote or right angle bracket, your list comes up empty.
...roboticus
When your only tool is a hammer, all problems look like your thumb.
|
---|