cwchang has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to get the daily option data from cboe (http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx). Basically, you fill the form with the ticker, it returns a file. In the browser, you will be prompted to save it somewhere. To do this in perl, I tried to use WWW::Mechanize to fill out the form. However, I don't know how to catch the file returned. Here is my code: 
#!/usr/bin/perl -w
use strict;
$|++;

use WWW::Mechanize;
my $ticker = shift;

#UserAgent settings
my $agent = WWW::Mechanize->new();
push @{$agent->requests_redirectable}, 'POST';

$agent->cookie_jar(HTTP::Cookies->new(file => "../../run/lwpcookies.tx
+t", autosave => 1));

#Grabbing the front page
$agent->get( "http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx
+" );
$agent->success or die $agent->response->status_line;

#Login
$agent->form_name("QuoteTableDownload");
$agent->field("ticker", "INTC");
$agent->submit();

$agent->success or die $agent->response->status_line;

open (OUT, "> ./tmp.html");
print OUT $agent->content;
close OUT;

[download]
------------------------------------- Does anyone know how to automatically save the returned data file? Thanks a lot! J

Replies are listed 'Best First'.
Re: save file from POST form??
by tachyon (Chancellor) on Sep 30, 2004 at 02:23 UTC

    The owners of this page have gone to some effort to ensure people are not screen scraping their page using a robot. This involves lots of javascript. Interestingly they have the equivalent of fatalsToBrowser enabled.

    Here is one way to get the data by automating IE. This is only presented for interest and is a fragile solution. Use of such code may breach the cboe.com TOS/TOC. Usual suggestion is not to be a cheapskate and buy the data you want. 

    use Win32::IE::Mechanize;
use Win32::GuiTest qw(FindWindowLike SetForegroundWindow SendKeys);

my $ticker = 'INTC';
my $time = time();
my $filename = "$ticker-$time.txt";

my $ie = Win32::IE::Mechanize->new( visible => 1 );

$ie->get( "http://www.cboe.com/DelayedQuote/QuoteTableDownload.aspx" )
+;

if ( fork ) {
    $ie->form_name( "QuoteTableDownload" );
    $ie->set_fields( ticker => $ticker  ),
    $ie->click_button( value =>'Download' );
    print "Closing IE\n";
    $ie->close;
}
else {
    my $window;
    sleep 1 until ($window) = FindWindowLike(0, "^File Download" );
    print "Got $window\n";
    SetForegroundWindow($window);
    SendKeys("{TAB}~");
    sleep 1;
    SendKeys("$filename~");
}

    [download]

    cheers

    tachyon

[reply]
[d/l]

Back to Seekers of Perl Wisdom