Automating downloads with Perl

yakkers has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Automating downloads with Perl by mpeters (Chaplain) on Feb 21, 2005 at 14:55 UTC
If I understand you correctly, then the answer is a most definite yes! This is typically called screen scraping or spidering. The best tool I've found for the job is WWW::Mechanize. It's pretty self explanatory. If you want to create WWW:::Mechanize scripts automatically, and you know that the sites you are using are not dependent on Javascript to function, you can use HTTP::Recorder to create a proxy that will record your actions with a website so that you can repeat them. As far as code samples go, the above linked documentation provides lots of examples, but WWW::Mechanize::Examples is another good resource if you need it.	[reply]
Re: Automating downloads with Perl by gellyfish (Monsignor) on Feb 21, 2005 at 14:53 UTC
You probably want to use WWW::Mechanize /J\	[reply]
Re: Automating downloads with Perl by dragonchild (Archbishop) on Feb 21, 2005 at 15:04 UTC
Completely off-topic, but the node title "Can perl do it?" reminds me of the children's show Bob the Builder. He's constantly asking "Can we build it?" and the response is always Yes we can!. :-) Being right, does not endow the right to be rude; politeness costs nothing. Being unknowing, is not the same as being stupid. Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence. Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.	[reply]
Re^2: Automating downloads with Perl by Aristotle (Chancellor) on Feb 21, 2005 at 20:56 UTC
The zeroth rule of Perl club is, Perl can do it. Makeshifts last the longest.	[reply]
Re: Automating downloads with Perl by aufflick (Deacon) on Feb 22, 2005 at 00:10 UTC
The obviouis answer is that you need WWW::Mechanize to get the html, submit the form etc. But then you need to find the right links, etc. I recommend you also look at HTML::Tree which makes light work of plucking the right bits out of a complex html document. Don't be scared if it seems complex - it's really a quite simple matter of parsing the html, and using `my @bullets = $tree->look_down('_tag', 'li'); map { print $_->as_text; } @bullets;` [download] or something along those lines...	[reply] [d/l]
Re: Automating downloads with Perl by punkish (Priest) on Feb 21, 2005 at 21:29 UTC
can perl do it? Irrespective of what the definition of "it" is, the answer is "yes." I've used LWP::Simple for this kind of stuff and it works just swell. Here is some pseudo code -- `1 connect to the site 2 authenticate 3 move to the "certain area" 4 determine the freshness of the patches 5 download` [download] Heck, I just rewrote your original question. ;-) -- when small people start casting long shadows, it is time to go to bed	[reply] [d/l]
Re: Automating downloads with Perl by sh1tn (Priest) on Feb 21, 2005 at 23:44 UTC
`$_ = 'can Perl do it?' and #in case I dare to think s\|.+\|I see - TMTOWTDI\|;` [download]	[reply] [d/l]