Scrap shadow DOM website with Perl

vpperl has asked for the wisdom of the Perl Monks concerning the following question:

Hi all, My first post here, so hope I'm following all rules, as well as I'm not asking stipid questions :)

Guys, I'm trying to extract the content of a website, which is implemented with DOM objects, so the pure HTML parsing is not giving me any favor. For example, I'm trying to extract all player names from the following team:

https://www.fanteam.com/fantasy/dashboard/291187/5083071/19

Any suggestions/ideas/examples where to start with? I have some Perl experience, but haven't been able to do find anything that would help me with. Thanks!

Comment on Scrap shadow DOM website with Perl

Replies are listed 'Best First'.
Re: Scrap shadow DOM website with Perl by Corion (Patriarch) on Jan 20, 2021 at 13:45 UTC
Instead of scraping the page, I would look at decoding the Websocket communication that this site has with the backend.	[reply]
Re^2: Scrap shadow DOM website with Perl by vpperl (Novice) on Jan 20, 2021 at 13:50 UTC
That sound interesting, also having in mind that I would like to do this for more than one team. Any starting point with this idea?	[reply]
Re^3: Scrap shadow DOM website with Perl by Corion (Patriarch) on Jan 20, 2021 at 14:43 UTC
Personally, I like to look at the data first using the developer tools of Firefox or Chrome, and then implement the Websocket client using Mojo::UserAgent.	[reply]
Re^4: Scrap shadow DOM website with Perl by vpperl (Novice) on Jan 20, 2021 at 15:36 UTC
Re^5: Scrap shadow DOM website with Perl by vpperl (Novice) on Jan 21, 2021 at 13:21 UTC
Some notes below your chosen depth have not been shown here
Re: Scrap shadow DOM website with Perl by marto (Cardinal) on Jan 20, 2021 at 13:40 UTC
Do you mean player names as in the people playing the sport or those listed as playing the Fantasy game listed in the 'Names' column on the bottom left hand side?	[reply]
Re^2: Scrap shadow DOM website with Perl by vpperl (Novice) on Jan 20, 2021 at 13:49 UTC
Hi, I mean the football players on the right hand side in the "football field".	[reply]
Re^3: Scrap shadow DOM website with Perl by marto (Cardinal) on Jan 20, 2021 at 14:01 UTC
Using your browsers developer tools, refresh the page and do a search for one of the player surnames, it reveals how their API calls work and how the data is returned.	[reply]
Re^4: Scrap shadow DOM website with Perl by vpperl (Novice) on Jan 20, 2021 at 14:22 UTC
Re^5: Scrap shadow DOM website with Perl by Corion (Patriarch) on Jan 20, 2021 at 14:42 UTC
Some notes below your chosen depth have not been shown here