actionphotos has asked for the wisdom of the Perl Monks concerning the following question:

Hello, I am looking for a way to continuously (maybe every 10 minutes), scrape an ASP page to look for schedule changes where it can then notify me of the change. What is confusing me is that all pages show the same URL, so I am not sure how to specify a specific URL, or is it not possible? to see what I mean, go to http://www.nes.com/Events.asp?MO=2 you can then click on the Mite event, or Presidents day event etc.... once there clicking on the schedule button will show that events schedule. But I noticed that the URL when viewing the schedule is the same no matter which event I am viewing... is there a way to find out a specific URL? or some way to download to a file the schedule every x minutes to look for changes? thanks!! John

Replies are listed 'Best First'.
Re: scraping an ASP page
by huck (Prior) on Feb 21, 2017 at 20:52 UTC

    If you look at the page source you will find that page is full of javascript calls like 

    <td width=50% onMouseOver="this.bgColor='yellow'" onMouseOut="this.bgC
+olor='white'" onClick=goAction(2557131,2)><font color='Blue'>Mite Inv
+itational Tournament</font></td>

    [download]

    Sometimes it is easy to figure out what the javascript does, sometimes it is not so easy

    in this case goAction(2557131,2) SEEMS to submit a form POST event to "Events.asp?MO=2", with variables txtEvent=2557131 and txtAction=2, because it is a POST event these do not show up in the URL

    you could try to create a curl or LWP request to simulate that posted form

[reply]
[d/l]
Re: scraping an ASP page
by haukex (Archbishop) on Feb 22, 2017 at 07:54 UTC

    Hi actionphotos,

    huck noted the page contains JavaScript, which you'd need to execute to get the page to behave the same way it does in the browser. You may want to look into WWW::Mechanize::Firefox.

    Hope this helps,
    -- Hauke D

[reply]
      thank you. I am not familiar with that module. Would you happen to have an example to help guide me in the right direction? I will start playing with it as well... see where I can get. thanks again!
[reply]
[reply]

Back to Seekers of Perl Wisdom