I want to be able to go to the following web site periodically and pull back the data. http://emma.msrb.org/MarketActivity/RecentTrades.aspx This seems simple enough yet I'm having an issue in that a get() only gets the page where it tells me i have to agree to terms and conditions. I know it then gives me a cookie and I'm good to go. I want to be able to get that cookie programatically and be able to go to the site. I think I should be able to do this. I've tried both the CGI methods and the HTTP:COOKIES methods but I can't get much more than the first page no matter what I do. I definitely feel it has to do with the cookie. I'd post my code but it's really ugly at this point with all the comments... is there something I'm fundamentally missing here? here's the code anyway... like I said, it's ugly and missing a lot of stuff 
#!/apps/VA/perl5.8.0/bin/perl
use Net::SMTP;
use Sys::Hostname;
use HTTP::Cookies;
use HTTP::Request::Common;
use Time::Local;
use POSIX (strftime);
use LWP::UserAgent;
use LWP::Simple;
use Getopt::Std;
use LWP::Simple qw( $ua get );

$ua->proxy( 'http', 'http://http.proxy.fmr.com:8000' );
# Load LWP class for "cookie jar" objects
#$ua->cookie_jar( { file=> "emma_cookie.txt" });

#my $browser = LWP::UserAgent->new( );
my $cookie_jar = HTTP::Cookies->new(
 'file' => 'emma_cookie.txt'
);
#$browser->cookie_jar( $cookie_jar );

#$cookies->set_cookie(0,'cookiename', 'value','/','emma.msrb.org',80,0
+,0,86400,0);
#$ua->cookie_jar($cookies);
# Now make your request
my $request = 'http://emma.msrb.org/MarketActivity/RecentTrades.aspx';
print "$request\n";
$cookie_jar->add_cookie_header( $request );
#getstore('http://emma.msrb.org/MarketActivity/RecentTrades.aspx', "MS
+RB.txt");
#getstore('http://www.nasdaqtrader.com/dynamic/SymDir/otherlisted.txt'
+, "otherlisted.txt");
#getstore('http://www.nasdaqtrader.com/dynamic/SymDir/nasdaqlisted.txt
+', "nasdaqlisted.txt");

#my $ua = LWP::UserAgent->new;
#$ua->cookie_jar($cookie_jar);

#my $request = GET "http://www.emma.msrb.org"; # don't worry, it doesn
+'t get saved :-)
#my $response = $ua->request( $request );

#$cookie_jar->extract_cookies( $response );

[download]

In reply to Getting data from an ASPX generated web page by billycote

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.