Re: problem parsing html

My experience with HTML::TreeBuilder showed me that I should use HTML::TreeBuilder::XPath which of course I would recommend you use also.

( HTML::TreeBuilder is nice but HTML::TreeBuilder::XPath is sufficiently abstract in order to be elegant )

Let's see how your code would look if you would use above mentioned module :

  1  #!/usr/local/bin/perl
  2  use strict;
  3  use warnings;
  4  use LWP::Simple;
  5  use HTML::TreeBuilder::XPath;
  6  use Data::Dumper;
  7  use feature 'say';
  8  my $url="http://www.ebi.ac.uk/thornton-srv/databases/cgi-bin/pdbs
+um/GetPage.pl?pdbcode=1r9t&template=main.html";
  9
 10  my $p = HTML::TreeBuilder::XPath->new_from_content(get($url));
 11
 12  my @chain_tags = $p->findnodes("//td//a[contains(\@href,'chain=')
+]");
 13  my @chains = map { $_->attr('href') =~ /chain=(\w)/ } @chain_tags
+;
 14  say " Number of chains : " . scalar @chains;
 15  say @chains;
[download]

EDIT:small adjustments

OUTPUT:

 
Number of chains : 11
AABCEFHIJKL
[download]

First of all we have simplified the code from 33 lines to 15 lines. Second , we maintained the meaning of the code , which was

"Give me the a tags which have attribute href which matches regex "&chain=(\w)" so that the a tags have a parent tag td. Now take those a tags and apply a regex on their href attribute and take the word after the chain= substring".

That is exactly what this XPath query says => //td//a[contains(\@href,'chain=')]

I have also used this Firefox addon to check that my XPath was right.

If you're interested in reading more about XPath read here and here.

You also have a mistake in your code , you delete the HTML::TreeBuilder object after the first iteration of the for loop.

Comment on Re: problem parsing html Select or Download Code

Replies are listed 'Best First'.
Re^2: problem parsing html by missingthepoint (Friar) on Jul 16, 2009 at 09:09 UTC
I find using CSS selectors easier, in which case you stick with HTML::TreeBuilder::XPath and throw in HTML::Selector::XPath. Then: `my $selector = "#fiddlesticks li a"; my $tree = HTML::TreeBuilder::XPath->new(); my $xpath = HTML::Selector::XPath->new($selector)->to_xpath; my @nodes = $tree->findnodes($xpath); for my $n (@nodes) { print "Fiddlesticks Link: ", $n->attr('href'), "\n"; }` [download] The zeroeth step in writing a module is to make sure that there isn't already a decent one in CPAN. (-- Pod::Simple::Subclassing)	[reply] [d/l]
Re^2: problem parsing html by paola82 (Sexton) on Jul 16, 2009 at 08:37 UTC
thanks...you quite solved my problem..but I have the last question...:the script returned chains...but they are as a block...I need to ave A,B,C etc...as single element of the array...because I need to start a cycle for every chain...I try to search "split" the name make me remember something...that can be useful?	[reply]
Re^3: problem parsing html by spx2 (Deacon) on Jul 16, 2009 at 09:00 UTC
As I understand , you only want unique chains , you can do this with minimum effort: 1 #!/usr/local/bin/perl 2 use strict; 3 use warnings; 4 use LWP::Simple; 5 use HTML::TreeBuilder::XPath; 6 use Data::Dumper; 7 use feature 'say'; 8 my $url="http://www.ebi.ac.uk/thornton-srv/databases/cgi-bin/pdbsu +m/GetPage.pl?pdbcode=1r9t&template=main.html"; 9 10 my $p = HTML::TreeBuilder::XPath->new_from_content(get($url)); 11 12 my @chain_tags = $p->findnodes("//td//a[contains(\@href,'chain=')] +"); 13 my $uniq_chains = { map { $_->attr('href') =~ /chain=(\w)/; ($1,1) +; } @chain_tags }; 14 say " Number of chains : " . scalar keys %$uniq_chains; 15 say keys %$uniq_chains; [download]	[reply] [d/l]
SOLVED: problem parsing html by paola82 (Sexton) on Jul 16, 2009 at 10:10 UTC
I used your code and add the function split #!/usr/local/bin/perl use strict; use warnings; use LWP::Simple; use HTML::TreeBuilder::XPath; use Data::Dumper; my $url="http://www.ebi.ac.uk/thornton-srv/databases/cgi-bin/pdbsum/Ge +tPage.pl?pdbcode=1r9t&template=main.html"; my $p = HTML::TreeBuilder::XPath->new_from_content(get($url)); my @chain_tags = $p->findnodes("//td//a[contains(\@href,'chain=')]"); my @chains = map { $_->attr('href') =~ /chain=(\w)/ } @chain_tags; my @chain = map {split ' '} @chains; foreach my $chain (@chain){print "$chain\n";} [download] the output was: `pisolo@pisolo:~/Scrivania/test$ perl test7.pl A A B C E F H I J K L` [download] so I think I can iterate my analysis for every chain....using as array with foreach....Thanks.... :-)	[reply] [d/l] [select]


Keep It Simple, Stupid
	PerlMonks