in reply to Re^2: Web scraping
in thread Web scrapping

Is this "AI" generated code? It looks like something it would come up with since it doesn't understand what the problem is, or have any domain knowledge. The content of the page is constructed after it's loaded. The table you're trying to parse simply doesn't exist at the point your code runs. The data 'is' in the page, see cjs.initialFeeds['results'], however this is very fragile and subject to change, even automatically. As usual 1nickt has given a sensible response, you could also use a headless browser such as WWW::Mechanize::Chrome to automate an actual browser to visit the page, waiting for everything to actually happen and acquiring the data via the query mechanisms it provides, or passing the complete HTML to your Mojo::DOM code. The sensible option is the free APIs already provided, rather than playing whack a mole with this other site.

Replies are listed 'Best First'.
Re^4: Web scraping
by joyfedl (Acolyte) on Jul 11, 2025 at 19:41 UTC

    its not AI generated

    this is the reference where been learning how to web scrap and took so +me examples from https://brightdata.com/blog/web-data/web-scraping-wi +th-perl

    you can see that my code is similar to the examples on that web

      While this page leaves a lot to be desired, the section Scraping Dynamic Websites does touch on the unreliability of basing parsing code on 'view source', since there's a great deal more to consider (previously mentioned domain specific knowledge, MDN Web Docs is a good starting point). I still automate scraping frequently, but only where there's no alternative sensible data source, see previous posts. If you can write it well, and once then this makes for a happier life than playing Whac-A-Mole with a sites such as this which frequently change their layout, html IDs and so fourth.