Fetching titles and error handling

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I have made a script that fetches the title of a (supposed) web page. The problem arises when I give a big file (which is n't even text/html) to the script. The file gets downloaded (waste of bandwidth) and after downloading it, the script notices it isn't text/html. Here's the script:

#!/usr/bin/perl -w

use HTML::TokeParser;
use LWP::Simple;
use Encode;

sub getTitle {
    my $stream = HTML::TokeParser->new(@_);
    if(defined $stream->get_tag("title")) {
        my $title = $stream->get_trimmed_text;
        return encode_utf8($title);
    }
}

my $browser;
my $url = $ARGV[0];

BEGIN {
    use LWP::UserAgent;
    $browser = LWP::UserAgent->new;
    $browser->agent("Mozilla/5.0");
    $browser->timeout(15);
}

my $resp = $browser->get($url);

die "Error getting $url: ", $resp->status_line, "\n"
    unless $resp->is_success;

die "Not HTML, it's ", $resp->content_type, "\n"
    unless $resp->content_type eq 'text/html';

if(my $title = getTitle($resp->content_ref)) {
    print "Title: '$title'\n";
}
[download]

What would be the solution for this? I should probably make a limit of some sort, say 32kB, for the get(). This seems to be a common problem, but I haven't found a decent solution. Also, should I worry about the content-encodings of the pages or does encode_utf8() take care of those too? Thanks in advance.

Comment on Fetching titles and error handling Download Code

Replies are listed 'Best First'.
Re: Fetching titles and error handling by duelafn (Parson) on Jul 01, 2009 at 10:29 UTC
Download the headers first `my $resp = $browser->head($url);` then decide whether to get the rest after considering the content type and size. Good Day, Dean	[reply] [d/l]
Re: Fetching titles and error handling by bart (Canon) on Jul 01, 2009 at 10:49 UTC
LWP::UserAgent provides a mechanism to download a file with a callback for the processing. You could do your check in the callback for the first buffer (although a check for the content-type in the headers is a very sane idea, too), and try to pull out the title in this or one of the following buffers. Unfortunately I don't think there's an easy way to treat the buffer data from the callbacks as an input stream for HTML::TokeParser::Simple, which seems like the most intuitive approach to me. Sometimes I'm really jealous of the features in other languages. (coroutines, iterators)	[reply]
Re: Fetching titles and error handling by Anonymous Monk on Jul 01, 2009 at 11:55 UTC
http://search.cpan.org/grep?cpanid=GAAS&release=libwww-perl-5.828&string=title&i=1&n=1&C=0 Read more... LWP::Simple version (919 Bytes) WWW::Mechanize `#!/usr/bin/perl -- use strict; use warnings; #use open ':locale'; use open ':encoding(UTF-8)'; die "\nUsage: $0 url\n" unless @ARGV; main(@ARGV); sub main { my ($url) = @_; use WWW::Mechanize 1.54; my $ua = WWW::Mechanize->new( agent => "Mozilla/5.0", timeout => 15, ); $ua->get($url); die "Not HTML, it's ", $ua->ct, "\n" unless $ua->is_html(); if ( my $title = $ua->title ) { print "Title: '$title'\n"; } }` [download] Read more... Sample output (465 Bytes)	[reply] [d/l] [select]
Re: Fetching titles and error handling by Anonymous Monk on Jul 01, 2009 at 12:08 UTC
Thanks a lot! That works perfectly.	[reply]