HTML::TokeParser token dumper

When you wish to parse a HTML document (or a look alike) using HTML::TokeParser, and you're not quite sure how the document will be tokenized, this snippet can give you a good idea of what's going on.

To run, simply type "perl scriptname.pl url" you'll a dump on STDOUT looking like:

TYPE : D
####
0:<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
>
####
1:
####
####################################################

TYPE : T
####
0: 

####
1:
####
2:
####
####################################################

TYPE : C
####
0:<!--took this out for IE6ites  "http://www.w3.org/TR/REC-html40/loos
+e.dtd"-->
####
1:
####
####################################################

TYPE : T
####
0:

####
1:
####
2:
####
####################################################

TYPE : S
####
0:html
####
1:HASH(0x1afe3f4)
####
2:ARRAY(0x1afe40c)
####
3:<HTML>
####
4:
####
####################################################
[download]

#!/usr/bin/perl -w
use strict;
use LWP::Simple;
use HTML::TokeParser;

my $url = shift or die "usage: ". __FILE__ ." url";
my $rawHTML = get($url); # attempt to d/l the page to mem

die "LWP::Simple messed up $!" unless ($rawHTML);

my $tp;
$tp = HTML::TokeParser->new(\$rawHTML) or die "WTF $tp gone bad: $!";

# And now -- a generic HTML::TokeParser loop

while (my $token = $tp->get_token)
{
    my $ttype = shift @{ $token };
    print "TYPE : $ttype\n####\n";
    printf( join( '',
                  map { "$_:%s\n####\n" } 0..@{$token}
                 )
            ,
            @{$token}
          );
    print "####################################################\n\n";
}
[download]

Comment on HTML::TokeParser token dumper Select or Download Code

Replies are listed 'Best First'.
(crazyinsomniac): HTML::Parser token dumper by crazyinsomniac (Prior) on Dec 28, 2001 at 08:31 UTC
HTML::Parser version here (same usage): #!/usr/bin/perl -w use strict; use LWP::Simple; use HTML::Parser; my $url = shift or die "usage: ". __FILE__ ." url"; my $rawHTML = get($url); # attempt to d/l the page to mem my @parsed; die "LWP::Simple messed up $!" unless ($rawHTML); my %TPS =( api_version => 3, handlers => {default => [\@parsed, "event,text"]} ,); my $tp = HTML::Parser->new( %TPS ); $tp->parse($rawHTML); for my $token(@parsed) { my $ttype = shift @{ $token }; print "TYPE : $ttype\n####\n"; printf( join( '', map { "$_:%s\n####\n" } 0..@{$token} ) , @{$token} ); print "####################################################\n\n"; } __END__ TYPE : start_document #### 0: #### 1: #### #################################################### TYPE : declaration #### 0:<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" > #### 1: #### #################################################### TYPE : text #### 0: #### 1: #### #################################################### TYPE : comment #### 0:<!--took this out for IE6ites "http://www.w3.org/TR/REC-html40/loos +e.dtd"--> #### 1: #### #################################################### [download] ___crazyinsomniac_______________________________________ `Disclaimer: Don't blame. It came from inside the void` `perl -e "$q=$_;map({chr unpack qq;H;,$_}split(q;;,qH*));print;$q/$q;"`	[reply] [d/l]

Replies are listed 'Best First'.

(crazyinsomniac): HTML::Parser token dumper
by crazyinsomniac (Prior) on Dec 28, 2001 at 08:31 UTC

#!/usr/bin/perl -w
use strict;
use LWP::Simple;
use HTML::Parser;

my $url = shift or die "usage: ". __FILE__ ." url";
my $rawHTML = get($url); # attempt to d/l the page to mem
my @parsed;
die "LWP::Simple messed up $!" unless ($rawHTML);
my %TPS =( api_version => 3,
          handlers    => {default => [\@parsed, "event,text"]}
         ,);
   
my $tp = HTML::Parser->new( %TPS );
$tp->parse($rawHTML);

for my $token(@parsed)
{
    my $ttype = shift @{ $token };
    print "TYPE : $ttype\n####\n";
    printf( join( '',
                  map { "$_:%s\n####\n" } 0..@{$token}
                 )
            ,
            @{$token}
          );
    print "####################################################\n\n";
}


__END__
TYPE : start_document
####
0:
####
1:
####
####################################################

TYPE : declaration
####
0:<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
>
####
1:
####
####################################################

TYPE : text
####
0: 

####
1:
####
####################################################

TYPE : comment
####
0:<!--took this out for IE6ites  "http://www.w3.org/TR/REC-html40/loos
+e.dtd"-->
####
1:
####
####################################################
[download]

___crazyinsomniac_______________________________________
Disclaimer: Don't blame. It came from inside the void
perl -e "$q=$_;map({chr unpack qq;H*;,$_}split(q;;,q*H*));print;$q/$q;"

[reply]
[d/l]