Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re^3: WWW::Mechanize::Chrome VERY slow on xpath obtaining TDs of a TR (updated)

by LanX (Sage)
on Nov 27, 2022 at 10:38 UTC ( #11148403=note: print w/replies, xml ) Need Help??


in reply to Re^2: WWW::Mechanize::Chrome VERY slow on xpath obtaining TDs of a TR
in thread WWW::Mechanize::Chrome VERY slow on xpath obtaining TDs of a TR

That's one approach.

But as I said I think putting the logic into a more elaborate xpath to do the heavy lifting inside the browser would fix your performance issue without needing HTML::Tree

IMHO your code will force the Perl part in W:M:C to do a lot of own filtering and create thousands of proxy objects. These Perl objects will also tunnel requests back and forth to the browser for most method calls.

Hence many potential bottlenecks.

update

as an illustration, this xpath in chrome's dev console for https://meta.wikimedia.org/wiki/Wikipedia_article_depth returns 1016 strings at once

//table[3]//tr//td//text()

Disclaimer: I don't have W:M:C installed and my xpath foo is rusted, so I'm pretty sure there are even better ways to do it.

Cheers Rolf
(addicted to the 𐍀𐌴𐍂𐌻 Programming Language :)
Wikisyntax for the Monastery

  • Comment on Re^3: WWW::Mechanize::Chrome VERY slow on xpath obtaining TDs of a TR (updated)
  • Download Code

Replies are listed 'Best First'.
Re^4: WWW::Mechanize::Chrome VERY slow on xpath obtaining TDs of a TR (updated)
by ait (Hermit) on Nov 29, 2022 at 09:15 UTC

    True.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11148403]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (3)
As of 2023-01-30 04:29 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?