Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I'm running this code - 
use strict;
use warnings;

use HTML::Parse;
use HTML::FormatText;
use LWP::Simple;

my $url = "http://www.perlmonks.org";
my $html = get($url);

defined $html or die "Can't fetch HTML from: ",$url;

my $ascii = HTML::FormatText->new->format(parse_html($html));
print $ascii;

[download]

It gives me the following output - 
[TABLE NOT SHOWN][TABLE NOT SHOWN][TABLE NOT SHOWN][TABLE NOT SHOWN]

   PerlMonks lovingly hand-crafted by Tim Vroom.
   PerlMonks went on a couple dates, and then decided to shack up with
   The Perl Foundation.
   Wonderful Web Servers and Bandwidth Generously Provided by pair
   Networks

[download]
I'm running the code on Mac OS X; Perl, v5.8.8 built for darwin-thread-multi-2level (with 4 registered patches).

Can anyone suggest what I might do to extract the raw text from a website since this method is clearly not working for me. I will be running this code on a server, linux hopefully, when I get up and running.

Thanks!

Replies are listed 'Best First'.
Re: Extracting raw text from a website
by Your Mother (Archbishop) on Dec 03, 2009 at 21:22 UTC

    XML::LibXML version- 

    use strict;
use warnings;
use XML::LibXML;
use LWP::Simple;

my $url = "http://www.perlmonks.org";
my $html = get($url);
my $parser = XML::LibXML->new;
$parser->recover_silently(1);
$parser->keep_blanks(1);

my $doc = $parser->parse_html_string($html);

print $doc->textContent;

    [download]

    See also: Re: Strip HTML tags again and the follow-up.

[reply]
[d/l]
Re: Extracting raw text from a website
by CountZero (Bishop) on Dec 03, 2009 at 20:32 UTC
    What exactly do you mean with "raw text"? If you want to strip all HTML-markup from the page, try HTML::Strip.

    CountZero

    A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

[reply]
      Hi! I mean what the user sees. Or put another way, the text that a "talking browser" would read to me if I were blind. Thanks!
[reply]
        HTML::Strip will do that for you:
        use strict;
use warnings;

use LWP::Simple;
use HTML::Strip;

my $url = "http://www.perlmonks.org";
my $html = get($url);

defined $html or die "Can't fetch HTML from: ",$url;

my $hs = HTML::Strip->new();

my $clean_text = $hs->parse( $html );
$hs->eof;

print $clean_text;

        [download]
        Deleting the spurious whitespace which this module adds, is left as an exercise for the reader.

        CountZero

        A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James

[reply]
[d/l]
        A reply falls below the community's threshold of quality. You may see it by logging in.

Back to Seekers of Perl Wisdom