Best way to parse HTML returned from secure server

ShayShay has asked for the wisdom of the Perl Monks concerning the following question:

Hi all...

I'm trying to write a script that will parse HTML that is returned from a secure server. No username or password is necessary. It does use a session variable and requires javascript to be enabled in order to spit out an HTML page other than "Please enable JavaScript in your browser."

I've installed Crypt and it's working fine. But... I'm not sure how to handle the session variable and I have no clue how to get beyond the javascript error.

I'd appreciate any insight anyone can provide. Thanks!

Comment on Best way to parse HTML returned from secure server

Replies are listed 'Best First'.
Re: Best way to parse HTML returned from secure server by kyle (Abbot) on Jan 30, 2008 at 17:39 UTC
Super Search is your friend. Here are a few of the things I found with it: How to login to a form which has Javascript OnSubmit method usirng Perl crawling past a javascript submit function WWW::Mechanize and fooling server for javascript	[reply]
Re^2: Best way to parse HTML returned from secure server by ShayShay (Acolyte) on Jan 30, 2008 at 20:07 UTC
Thanks, but I've looked through those (and more)... and nothing seems to work. What I'm looking for is something to get past the noscript tags.	[reply]
Re: Best way to parse HTML returned from secure server by GrandFather (Saint) on Jan 30, 2008 at 20:26 UTC
Can you provide a striped down version of the page (in code tags) or a link to a site that demonstrates the problem. From your scanty description it seems likely that the JavaScript provides a signature of some sort to the server so that the server side code knows that JavaScript is enabled. That may be a tricky problem to solve! Perl is environmentally friendly - it saves trees	[reply]
Re^2: Best way to parse HTML returned from secure server by ShayShay (Acolyte) on Jan 30, 2008 at 20:38 UTC
Since the page is public anyway... here's the URL: https://admin8.gtc.edu/wa/wa?&TOKENIDX=3292319802&CONSTITUENCY=WBST&type=M&pid=ST-XWBSTS I want to be able to go through to the "Search for Classes" link. Then I want to be able to tell the form to search for something specific, return the results, parse with perl, and include just the specific information I asked for... like Course Name or anything like that. I'm sorry it is complicated. I'm newish to Perl.	[reply]