Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Automating downloads with Perl

by yakkers (Initiate)
on Feb 21, 2005 at 14:47 UTC ( [id://433062]=perlquestion: print w/replies, xml ) Need Help??

yakkers has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I have a windows pc connected to the ineternet. I want to write a program which goes to a site, enters userid and password, goes to a certain area and downloads the latest patches. Can this be automated via perl? If so can someone point me in the right direction to the commands i would use. Thanks

Retitled by davido from 'can perl do it?'.

Replies are listed 'Best First'.
Re: Automating downloads with Perl
by mpeters (Chaplain) on Feb 21, 2005 at 14:55 UTC
    If I understand you correctly, then the answer is a most definite yes! This is typically called screen scraping or spidering. The best tool I've found for the job is WWW::Mechanize. It's pretty self explanatory.

    If you want to create WWW:::Mechanize scripts automatically, and you know that the sites you are using are not dependent on Javascript to function, you can use HTTP::Recorder to create a proxy that will record your actions with a website so that you can repeat them.

    As far as code samples go, the above linked documentation provides lots of examples, but WWW::Mechanize::Examples is another good resource if you need it.
Re: Automating downloads with Perl
by gellyfish (Monsignor) on Feb 21, 2005 at 14:53 UTC
Re: Automating downloads with Perl
by dragonchild (Archbishop) on Feb 21, 2005 at 15:04 UTC
    Completely off-topic, but the node title "Can perl do it?" reminds me of the children's show Bob the Builder. He's constantly asking "Can we build it?" and the response is always Yes we can!. :-)

    Being right, does not endow the right to be rude; politeness costs nothing.
    Being unknowing, is not the same as being stupid.
    Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
    Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

      The zeroth rule of Perl club is, Perl can do it.

      Makeshifts last the longest.

Re: Automating downloads with Perl
by aufflick (Deacon) on Feb 22, 2005 at 00:10 UTC
    The obviouis answer is that you need WWW::Mechanize to get the html, submit the form etc.

    But then you need to find the right links, etc. I recommend you also look at HTML::Tree which makes light work of plucking the right bits out of a complex html document. Don't be scared if it seems complex - it's really a quite simple matter of parsing the html, and using

    my @bullets = $tree->look_down('_tag', 'li'); map { print $_->as_text; } @bullets;
    or something along those lines...
Re: Automating downloads with Perl
by punkish (Priest) on Feb 21, 2005 at 21:29 UTC
    can perl do it?

    Irrespective of what the definition of "it" is, the answer is "yes."

    I've used LWP::Simple for this kind of stuff and it works just swell. Here is some pseudo code --

    1 connect to the site 2 authenticate 3 move to the "certain area" 4 determine the freshness of the patches 5 download

    Heck, I just rewrote your original question. ;-)

    --
    when small people start casting long shadows, it is time to go to bed
Re: Automating downloads with Perl
by sh1tn (Priest) on Feb 21, 2005 at 23:44 UTC
    $_ = 'can Perl do it?' and #in case I dare to think s|.+|I see - TMTOWTDI|;


Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://433062]
Approved by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having a coffee break in the Monastery: (3)
As of 2024-03-29 01:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found