Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re: Extracting HTML content between the h tags

by Anonymous Monk
on Aug 05, 2012 at 12:37 UTC ( [id://985521]=note: print w/replies, xml ) Need Help??


in reply to Extracting HTML content between the h tags

perl htmltreexpather.pl fudge.html

Hmm, so I would use the stack approach, ie *find*

q{ //div[@id='bodyContent']/* }

everything before first h2 tag is key/value pairs

after that , each h2 tag is the key , and the non-h2 tags that follow are the value

#!/usr/bin/perl -- use strict; use warnings; use HTML::TreeBuilder::XPath; my $page = q{<html> <head></head> <body> <div id="bodyContent"> <!-- start content --> <p>Key words: Some words. </p> <p>Date: 2012-01-16 </p> <p>Actualised: 2008-01-08 </p> <p>Commented: 05.06.2007 </p> <p>Encoded: Some code. </p> <h2> <span class="mw-headline" id="Problem"> Problem </span></h2 +> <p>Problem description. </p> <p>Another description. </p> <h2> <span class="mw-headline" id="Solution1"> Solution 1 </span +></h2> <p>Solution description. </p> <h2> <span class="mw-headline" id="Solution2"> Solution 2 </span +></h2> <p>Solution description. </p> <h2> <span class="mw-headline" id="Comment"> Comment. </span></h +2> <p>Text of the comment. </p> <p> <br/> </p> </div> <hr/> </body> </html>}; my $p = HTML::TreeBuilder::XPath->new_from_content( $page ); { my @nodes = $p->findnodes( q{//div[@id='bodyContent']/*}); use List::AllUtils qw( before ); my @before_h2 = before { $_->tag eq 'h2' } @nodes; splice @nodes, 0, scalar( @before_h2 ); my %body = map { split ':', $_->as_trimmed_text, 2 } @before_h2; while( @nodes ){ my $key = shift(@nodes)->as_trimmed_text; while( @nodes and $nodes[0]->tag ne 'h2' ){ my $val = shift(@nodes)->as_trimmed_text; $body{ $key } .= $val; } } use Data::Dump; dd\%body; } __END__ { "Actualised" => " 2008-01-08", "Comment." => "Text of the comment.", "Commented" => " 05.06.2007", "Date" => " 2012-01-16", "Encoded" => " Some code.", "Key words" => " Some words.", "Problem" => "Problem description.Another description.", "Solution 1" => "Solution description.", "Solution 2" => "Solution description.", }

Replies are listed 'Best First'.
Re^2: Extracting HTML content between the h tags
by Anonymous Monk on Aug 05, 2012 at 12:53 UTC
    The   "Comment." key stuck out, so a better idea might be to use the @id attribute as key
    my $key = shift(@nodes)->findvalue('*[@id]/@id');
      Thank you very much!
      Just tried the both approaches, it works even if the last h2-tag is missing ( appears in about 10 pages from > 400, for which I used the following workaround:
      my @solution_2 = $content->findvalues( './h2[4]/preceding-sibling::*' +); unless ( @solution_2 ) { @solution_2 = $content->findvalues( '//hr/preceding-sibling::*' ); }
      ... with substr as before ...
      Fortunately they have only one hr-tag in the page :-)
      With your approach it is not necessary anymore.
      BTW the content after the <h2>[4] is not important.
      Thanks again!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://985521]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others wandering the Monastery: (4)
As of 2024-03-28 21:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found