ASP's with lwp

ted@ki5u.net has asked for the wisdom of the Perl Monks concerning the following question:

Active Perl 5.8.2, Win 2K Server,
I'm bringing web pages to extract data using lwp::simple. Most pages retrieve perfectly. But when I access an active server page (http://www.cboe.com/mktdata/mktstat.asp), I'm getting the following error:

<h2>There was an error loading the header for this page.</h2>
<p> An error has occurred. You may contact your system administrator w
+ith the following information: </p>
<p>Location : /mktdata/mktstat.asp</p>
<p>Source : Microsoft VBScript runtime error</p>
<p>Code : 5</p>
<p>Description : Invalid procedure call or argument</p>
<br></br>
[download]

What do I need to do to access asp pages properly?

Thanks,
Ted

20040409 Edit by Corion: Fixed formatting

Comment on ASP's with lwp Download Code

Replies are listed 'Best First'.
Re: ASP's with lwp by Jenda (Abbot) on Apr 09, 2004 at 20:05 UTC
What do I need to do to access asp pages properly? Access only pages without bugs. They aparently have a bug in the page. Seems like they try to emit different output based on the user's browser and the code is broken. If I download the page with Opera set to indentify itself as MSIE everything is cool, if I change it to identify as Opera I get exactly the same error as you. Contact the site and inform them of the problem. (It might be better to talk about Opera than about your script. That way they can test it themselves and cannot say it's your bug.) Ask LWP to identify itself as MSIE. `use LWP::Simple qw($ua); $ua->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; .NET CL +R 1.0.3705; .NET CLR 1.1.4322)'); ...` [download] HTH, Jenda Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live. -- Rick Osborne Edit by castaway: Closed small tag in signature	[reply] [d/l]
Re: ASP's with lwp by esskar (Deacon) on Apr 09, 2004 at 16:24 UTC
try a different user-agent; on like the MS Internet Explorer is using. just a guess!	[reply]
Re: ASP's with lwp by eXile (Priest) on Apr 09, 2004 at 16:26 UTC
Hi, hard to tell what you need from the error message you got back. I think the ASP expects a certain line to be in your request. For instance if I access a page with Internet Explorer the header looks something like: `GET / HTTP/1.1 Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, applicati +on/msword, application/vnd.ms-excel, application/vnd.ms-powerpoint, a +pplication/x-shockwave-flash, / Accept-Language: nl Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) Host: www.emileaben.com Connection: Keep-Alive` [download] when I do the same in LWP it looks like: `GET / HTTP/1.1 TE: deflate,gzip;q=0.3 Connection: TE, close Host: www.emileaben.com User-Agent: libwww-perl/5.69` [download] You can make dumps like this with a packetsniffer (tcpdump, ethereal). My guess is you have to add/change your LWP request so it more closely resembles the Internet Explorer one. An obvious first try is to change the User-Agent header using the `agent` method of a LWP::UserAgent. Hope this helps.	[reply] [d/l] [select]