Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

problem parsing html

by paola82 (Sexton)
on Jul 15, 2009 at 13:36 UTC ( #780308=perlquestion: print w/replies, xml ) Need Help??

paola82 has asked for the wisdom of the Perl Monks concerning the following question:

Hi dear monks, I'm sorry to disturb you again...for problem solved before...but I have a long script...and I parsed as I did before but...now I don't know why it doesn't work...I'm not able to find the errors...the synthax seems ok...I'll paste an example code, because...the code is too long

#!/usr/local/bin/perl use strict; use warnings; use LWP::Simple; use HTML::TreeBuilder; my $url="http://www.ebi.ac.uk/thornton-srv/databases/cgi-bin/pdbsu +m/GetPage.pl?pdbcode=1r9t&template=main.html"; my $content=get ($url); my $chain; my @chains; my $p = HTML::TreeBuilder->new; $p->parse_content($content); my @tds = $p->look_down(_tag => q{td}); for my $td (@tds) { my $a = $td->look_down(_tag => q{a}); next unless $a; if ($a) { my $href = $a->attr(q{href}); if ($href =~ /&chain=(\w)"/) { print $1 qq{\n}; $chain=$1; push @chains, $chain; #I want to extract the chains +....A,B,C,D,E,F,H,I,J,K,L } } $p->delete; }

in this part I only want to print and push into an array the chains of the protein....so that my array would be @chains(A,B,C,etc....) but it doesn't work...If anyone could help me...please...I'm running a little crazy...thanks you all

Replies are listed 'Best First'.
Re: problem parsing html
by davorg (Chancellor) on Jul 15, 2009 at 13:46 UTC
      sorry I updated the correct url...that was the name of a pdb...19rt in the url insted of 1r9t
Re: problem parsing html
by spx2 (Deacon) on Jul 15, 2009 at 15:12 UTC

    My experience with HTML::TreeBuilder showed me that I should use HTML::TreeBuilder::XPath which of course I would recommend you use also.

    ( HTML::TreeBuilder is nice but HTML::TreeBuilder::XPath is sufficiently abstract in order to be elegant )

    Let's see how your code would look if you would use above mentioned module :

    1 #!/usr/local/bin/perl 2 use strict; 3 use warnings; 4 use LWP::Simple; 5 use HTML::TreeBuilder::XPath; 6 use Data::Dumper; 7 use feature 'say'; 8 my $url="http://www.ebi.ac.uk/thornton-srv/databases/cgi-bin/pdbs +um/GetPage.pl?pdbcode=1r9t&template=main.html"; 9 10 my $p = HTML::TreeBuilder::XPath->new_from_content(get($url)); 11 12 my @chain_tags = $p->findnodes("//td//a[contains(\@href,'chain=') +]"); 13 my @chains = map { $_->attr('href') =~ /chain=(\w)/ } @chain_tags +; 14 say " Number of chains : " . scalar @chains; 15 say @chains;

    EDIT:small adjustments

    OUTPUT:

    Number of chains : 11 AABCEFHIJKL

    First of all we have simplified the code from 33 lines to 15 lines. Second , we maintained the meaning of the code , which was

    "Give me the a tags which have attribute href which matches regex "&chain=(\w)" so that the a tags have a parent tag td. Now take those a tags and apply a regex on their href attribute and take the word after the chain= substring".

    That is exactly what this XPath query says => //td//a[contains(\@href,'chain=')]

    I have also used this Firefox addon to check that my XPath was right.

    If you're interested in reading more about XPath read here and here.

    You also have a mistake in your code , you delete the HTML::TreeBuilder object after the first iteration of the for loop.

      I find using CSS selectors easier, in which case you stick with HTML::TreeBuilder::XPath and throw in HTML::Selector::XPath. Then:

      my $selector = "#fiddlesticks li a"; my $tree = HTML::TreeBuilder::XPath->new(); my $xpath = HTML::Selector::XPath->new($selector)->to_xpath; my @nodes = $tree->findnodes($xpath); for my $n (@nodes) { print "Fiddlesticks Link: ", $n->attr('href'), "\n"; }

      The zeroeth step in writing a module is to make sure that there isn't already a decent one in CPAN. (-- Pod::Simple::Subclassing)
      thanks...you quite solved my problem..but I have the last question...:the script returned chains...but they are as a block...I need to ave A,B,C etc...as single element of the array...because I need to start a cycle for every chain...I try to search "split" the name make me remember something...that can be useful?
        As I understand , you only want unique chains , you can do this with minimum effort:
        1 #!/usr/local/bin/perl 2 use strict; 3 use warnings; 4 use LWP::Simple; 5 use HTML::TreeBuilder::XPath; 6 use Data::Dumper; 7 use feature 'say'; 8 my $url="http://www.ebi.ac.uk/thornton-srv/databases/cgi-bin/pdbsu +m/GetPage.pl?pdbcode=1r9t&template=main.html"; 9 10 my $p = HTML::TreeBuilder::XPath->new_from_content(get($url)); 11 12 my @chain_tags = $p->findnodes("//td//a[contains(\@href,'chain=')] +"); 13 my $uniq_chains = { map { $_->attr('href') =~ /chain=(\w)/; ($1,1) +; } @chain_tags }; 14 say " Number of chains : " . scalar keys %$uniq_chains; 15 say keys %$uniq_chains;
Re: problem parsing html
by Anonymous Monk on Jul 15, 2009 at 13:52 UTC
    "Doesn't work" isn't very specific :)

    Try adding

    warn "td $td"; warn "a $a";
    before next unless $a; and show what you get
      td HTML::Element=HASH(0x85a6a18) at test6.pl line 21. Use of uninitialized value in concatenation (.) or string at test6.pl line 22. a at test6.pl line 22.
        Switch with
        my( $a ) = $td->look_down(_tag => q{a});
Re: problem parsing html
by Anonymous Monk on Jul 15, 2009 at 15:25 UTC
    Here is your problem
    ||||||||||| / _____ ____ ||||||||||| / | | | | | | VVVVVVVVVVV / | | | | | | $p->delete; <----\ |-|--|---|----|--| ^^^^^^^^^^^ \ | \| |__| | ___| ||||||||||| \ ||||||||||| \
    you delete tree, NUTS!
Re: problem parsing html
by Anonymous Monk on Jul 15, 2009 at 15:08 UTC
    Instead of
    print $1 qq{\n}; $chain=$1; push @chains, $chain; #I want to extract the chains....A,B,C,D,E,F, +H,I,J,K,L
    you probably want
    print "chain = $1\n"; push @chains, $1;

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://780308]
Approved by moritz
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (6)
As of 2022-06-30 20:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My most frequent journeys are powered by:









    Results (98 votes). Check out past polls.

    Notices?