Seeking a more robust variant of HTML::TreeBuilder::XPath

Paradigma has asked for the wisdom of the Perl Monks concerning the following question:

Hello,

Using this component, sometimes happens I'm limited to fully parse some more complex or broken pages. Though my element is shown in browser's element viewer, I can't reach it in TreeBuilder's tree.

I would appreciate any other element, even commercial, conceptually similar or same to TreeBuilder, which would parse more aggressively pages with complex structure. Maybe the inability to parse fully could be resolved by tuning up with TreeBuilder initialization parameters?

Comment on Seeking a more robust variant of HTML::TreeBuilder::XPath

Replies are listed 'Best First'.
Re: Seeking a more robust variant of HTML::TreeBuilder::XPath by holli (Abbot) on Jul 12, 2019 at 12:44 UTC
Though my element is shown in browser's element viewer, I can't reach it in TreeBuilder's tree What you see in the browser via the developer tools and what you get when you fetch the same page (via curl, or LWP or whatever) must not necessarily be identical. Mainly because of Javascript possibly altering the site after load or something server side that looks at the user agent and is serving different content based on that. To check if that is the case look at the page source, not the current DOM. holli You can lead your users to water, but alas, you cannot drown them.	[reply] [d/l]
Re^2: Seeking a more robust variant of HTML::TreeBuilder::XPath by Fletch (Bishop) on Jul 12, 2019 at 18:53 UTC
If this is the case something like WWW::Selenium might be useful (in that it involves a "real" browser in the process and any JS logic can ~~wreak it's evil will upon~~ manipulate the contents). The cake is a lie. The cake is a lie. The cake is a lie.	[reply]
Re^3: Seeking a more robust variant of HTML::TreeBuilder::XPath by marto (Cardinal) on Jul 12, 2019 at 19:28 UTC
Also WWW::Mechanize::Chrome	[reply]
Re: Seeking a more robust variant of HTML::TreeBuilder::XPath by hippo (Archbishop) on Jul 12, 2019 at 12:33 UTC
Using this component, sometimes happens I'm limited to fully parse some more complex or broken pages. Those are 2 quite separate problems. If the pages are merely complex then HTML::TreeBuilder::XPath should parse them. You would be helping everyone, yourself included, if you could report such bugs to the maintainer (ideally with an SSCCE) so that they can be fixed. Try to ensure that you are posting the bug against the right dist - it may be that one of the dependencies is actually at fault. If the pages are broken then it's quite fair for HTML::TreeBuilder::XPath to fail to parse them. Instead you need a way to fix the page before parsing. Have you tried HTML::Valid?	[reply]
Re: Seeking a more robust variant of HTML::TreeBuilder::XPath by marto (Cardinal) on Jul 12, 2019 at 12:12 UTC
Perhaps Mojo::DOM would help, it deals with some insane things, broken HTML etc. Check it out, Super Search will find you many interesting uses. Update: feel free to provide some messed up example data, and what you're trying to parse out of it.	[reply]
Re: Seeking a more robust variant of HTML::TreeBuilder::XPath ( htmltreexpather , HTML::TreeBuilder::LibXML ) by Anonymous Monk on Jul 13, 2019 at 01:58 UTC
HTML::TreeBuilder::Xpath is plenty robust HTML::Tree is not your browser , its HTML::Tree Use htmltreexpather.pl to help you write more appropriate xpaths See Re: Issues using HTML::TreeBuilder::XPath and the HTML5 <section> tag ( ignore_unknown => 0 ), HTML::TreeBuilder::LibXML	[reply]