in reply to How can my script retrieve the contents of an existing webpage?

I would use the LWP:UserAgent for something like this. I've used this qute a few times to cache content from parts of a website - the syntax to get the slashdot page (for example) would be something like:

# create a user agent object use LWP::UserAgent; my $ua = new LWP::UserAgent; $ua->agent("AgentName/0.1 " . $ua->agent); my $url = "http://slashdot.org"; # Create a request my $req = new HTTP::Request GET => $url; $req->content_type('application/x-www-form-urlencoded'); $req->content('match=www&errors=0'); # Pass request to the user agent and get a response back my $res = $ua->request($req); # Check the outcome of the response if ($res->is_success) { print $res->content; } else { print "Was the URL correct?"; }

That's my two cents anyway.

  • Comment on Re: How can my script retrieve the contents of an existing webpage?
  • Download Code

Replies are listed 'Best First'.
Re: Answer: How can my script retrieve the contents of an existing webpage?
by merlyn (Sage) on Apr 03, 2001 at 18:55 UTC