Grabbing Web Page to process

dennis5775 has asked for the wisdom of the Perl Monks concerning the following question:

For some reason when this prints out I only get like the last third of the web site. Not sure why. I am assuming it is truncated in my variable. Can anyone explain what I am missing ?

 
#!/usr/bin/perl -w
use strict;
use warnings;
 require LWP::UserAgent;
 
 my $ua = LWP::UserAgent->new;
 $ua->timeout(10);
 $ua->env_proxy;
 
 my $response = $ua->get('http://www.herald-journal.com/government/pub
+licnotices/2015-legals/hj010515.html');
 
 if ($response->is_success) {
     print $response->decoded_content;  # or whatever
 }
 else {
     die $response->status_line;
 }
[download]

Comment on Grabbing Web Page to process Download Code

Replies are listed 'Best First'.
Re: Grabbing Web Page to process by Anonymous Monk on Jan 06, 2015 at 02:26 UTC
Why what do you get with lwp-download ? `$ lwp-download http://www.herald-journal.com/government/publicnotices/ +2015-legals/hj010515.html Saving to 'hj010515.html'... 27.6 KB received in 0 seconds (1.01e+003 KB/sec)` [download] Can anyone explain what I am missing ? You're using LWP the hard way ... without WWW::Mechanize :) Also, also item 4 from Basic debugging checklist use Data::Dump::dd instead of plain print for debugging All works the same for me `perl -MWWW::Mechanize -e " $ua = WWW::Mechanize->new; $ua->show_progress(1); $ua->get( shift ); print $ua->content; " -- http://www.herald-journal.com/government/publicnotices/2015-legals/hj010515.html`	[reply] [d/l] [select]
Re: Grabbing Web Page to process by locked_user sundialsvc4 (Abbot) on Jan 06, 2015 at 13:45 UTC
Definitely switch this code to use “good ol’ Mech.” There’s typically a lot of back-and-forth exchanges between client and server in retrieving any web-page ... handling cookies, maybe authentication, and so on ... and that module and its brethren know how to do all of these things already. (Yeah, I originally made the same mistake you did ...)
Re^2: Grabbing Web Page to process by Anonymous Monk on Jan 07, 2015 at 09:33 UTC
Hello mike robertson, don't be such a spammer, spamming people isn't cool ...	[reply]
Re^3: Grabbing Web Page to process by dennis5775 (Initiate) on Jan 07, 2015 at 18:13 UTC
thanks to everyone who made comments you are all life savers , thanks	[reply]