AI Cowboy has asked for the wisdom of the Perl Monks concerning the following question:
Greetings dear Monks,
I have been trying to, for my work project, get a Perl program that can automatically download all the packages linked to by a single Google web-page. Nothing nefarious of course, but there are literally hundreds of 20-100 MB sized files there, and I don't fancy doing this manually. I've actually been instructed to build such a program. The program isn't the hard part per se, as I've already completed the task in multiple ways, theoretically. The problem is, when I run even the following test code (with the extra "Use" statements left over from previous attempts):I get the following error in command prompt (I am using Windows 8, don't ask):#!/usr/bin/perl use LWP::UserAgent; use LWP::Simple; use URI::URL; use WWW::Mechanize; use HTML::LinkExtor; my $url = 'http://foo.bar.baz'; getprint('http://foo.bar.baz'); $user = LWP::UserAgent->new(); $user->get($url);
What am I doing wrong with my approach? Is there a way to fix/bypass this? Could a different programming language get the job done? I've gotten the script to download the files successfully (I tried on one of them manually with the script using LWP::Simple to save the file on my disk), but the page that links the downloads is unreadable apparently. UPDATE: I've tried wget, curl, and a few other things - even a method that worked yesterday to grab a test file off the net, and today the method I used to download a test file off the net with perl, doesn't work. Every time I use lwp to connect ANYWHERE on the net with perl now, it gives me the "500 Status Read Failed" error, "a non-blocking socket operation could not be completed immediately". I'm completely baffled by this. I can connect to local html files with lwp, but not anything on the internet, and I have no firewalls up.500 Status read failed: A non-blocking socket operation could not be c +ompleted i mmediately. <URL:http://foo.bar.baz>
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re: LWP and WWW:Mechanize not working (you think)
by Anonymous Monk on Jun 04, 2013 at 03:06 UTC | |
by AI Cowboy (Beadle) on Jun 04, 2013 at 03:15 UTC | |
by Anonymous Monk on Jun 04, 2013 at 03:25 UTC | |
by AI Cowboy (Beadle) on Jun 04, 2013 at 03:26 UTC | |
by Anonymous Monk on Jun 04, 2013 at 03:37 UTC | |
by AI Cowboy (Beadle) on Jun 04, 2013 at 03:51 UTC | |
by Anonymous Monk on Jun 04, 2013 at 04:04 UTC | |
| |
by Anonymous Monk on Jun 04, 2013 at 03:19 UTC |