Accessing and storing information from another website's database

sprakash has asked for the wisdom of the Perl Monks concerning the following question:

Hi All! Firstly, just wanted to wish you all a very Happy Holidays :-D. Secondly, I had a question -- I'm in the process of making a dynamic class scheduler and need to get the class information (days held, credits, etc) from another website -
https://spire.umass.edu/saha/guest/guest_frameset.asp
(if you click on "Class Search", then "Basic" and then "Search" for whatever class, you get the required results).
Thing is that is there any way to get this information through Perl? I seriously don't want to sit for a few days searching for all criteria and dumping it manually into a database. Additionally, the above site is very secure and don't allow external access to thier links, etc. I tried to request them to give me thier class database and they told me that they couldn't. Anyways, just neeeded some input and I look forward to hearing from you all. Take care, Surya

Comment on Accessing and storing information from another website's database

Replies are listed 'Best First'.
Re: Accessing and storing information from another website's database by derby (Abbot) on Dec 23, 2002 at 13:15 UTC
Hmmm ... it looks like it may be painful to code but WWW::Mechanize may be helpful. -derby	[reply]
Re: Accessing and storing information from another website's database by gjb (Vicar) on Dec 23, 2002 at 13:34 UTC
Most probably it doesn't really matter what you do in the browser window, just try and see what URL is used to retrieve the information and tamper with it to do the queries you want. It should be something along the lines of `http://www.somewhere.com/something.asp?class=someClass&mode=Basic` [download] . Now you can just fake this kind of request constructing it with LWP::UserAgent. In general, this is not hard. Good luck, -gjb-	[reply] [d/l]
Re: Re: Accessing and storing information from another website's database by sprakash (Acolyte) on Dec 23, 2002 at 13:39 UTC
gjb, Hi! Thanks for reply. I did try what you advise, but it seems that the website doesn't accept any direct URL ..... (1) it's HTTPS and (2) somehow its blocking for external refs to its information. Anyways, thanks for your help. Take care, Surya	[reply]
Re: Re: Re: Accessing and storing information from another website's database by hardburn (Abbot) on Dec 23, 2002 at 14:40 UTC
For blocking external refs, that is done by checking the HTTP Referer header. Try setting that portion to the header to whatever page the search normaly comes from. As for HTTPS, it looks like you'll need to do some tricks combining LWP::UserAgent and Crypt::SSLeay (I've never done HTTPS in perl, so I can't help you with the specifics).	[reply]
Re: Accessing and storing information from another website's database by Anonymous Monk on Dec 23, 2002 at 21:20 UTC
Thanks Guys ...... will play around with the stuff mentioned above ..... wish me the best of luck :-D. Thanks and have a Merry Christmas. Take care, Surya	[reply]