sstevens has asked for the wisdom of the Perl Monks concerning the following question:

I'm using the Live HTTP Headers extension on Firefox to figure out what is being sent in a javascript page. I have narrowed the headers down, and am now using the Live HTTP Replay part of the Headers extension. Here's what's in Replay:

POST: http://scores.covers.com/ajax/SportsDirect.Controls.LiveScoresCo +ntrols.Scoreboard,SportsDirect.Controls.LiveScoresControls.ashx?_meth +od=UpdateScoreboard&_session=no Host: scores.covers.com User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) +Gecko/20100115 Firefox/3.6 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0. +8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 115 Connection: keep-alive Referer: http://scores.covers.com/basketball-scores-matchups.aspx?t=0 Content-Length: 167 Content-Type: text/plain; charset=UTF-8 Cookie: WT_FPC=id=2a36d4f9a39ac80f0481267122335646:lv=1268419213631:ss +=1268419204254; __utma=75780999.682598437.1267111536.1268372315.12684 +08405.31; __utmz=75780999.1268372315.30.7.utmcsr=contests.covers.com| +utmccn=(referral)|utmcmd=referral|utmcct=/handicapping/experts/NBA-te +ams.aspx; CoversSignupReferrer=http://www.covers.com/index.aspx?t=0; +__utmc=75780999; ASP.NET_SessionId=5crxiiz4qrrvd445mkxtxxum; __utmb=7 +5780999.4.10.1268408405; LastDayViewed_9=2009-2010.2010.3.2; RefreshS +cores=12-2-2010-1268408433293 Pragma: no-cache Cache-Control: no-cache

And here's the POST content:

LeagueID=9\r\n GameDate=3-2-2010\r\n Season=2009-2010\r\n Refresh=true\r\n LastUpdateTime=01-01-1900\r\n type=Matchups\r\n RefreshStartTime=12-2-2010-1268408433293\r\n Week=\r\n conferenceID=

If I click the replay button, it brings up the data I need perfectly. So I've tried to copy this into a perl script, but I can't get it to work. Here is what I have:
#!/usr/bin/perl use strict; use HTTP::Request::Common; use LWP::UserAgent; my %fields = ( "LeagueID" => "9\r\n", "GameDate" => "3-2-2010\r\n", "Season" => "2009-2010\r\n", "Refresh" => "false\r\n", "LastUpdateTime" => "01-01-1900\r\n", "type" => "Matchups\r\n", "RefreshStartTime" => "11-2-2010-1268372427240\r\n", "Week" => "\r\n", "conferenceID" => "", ); my $robot = LWP::UserAgent->new; my $response = $robot->request(POST 'http://scores.covers.com/ajax/Spo +rtsDirect.Controls.LiveScoresControls.Scoreboard,SportsDirect.Control +s.LiveScoresControls.ashx?_method=UpdateScoreboard&_session=no', \%fi +elds); print $response->as_string;

This isn't working. It spits out the following:
HTTP/1.1 200 OK Cache-Control: private Date: Fri, 12 Mar 2010 16:04:06 GMT Content-Length: 309 Content-Type: text/html; charset=utf-8 Expires: Fri, 12 Mar 2010 16:04:06 GMT Client-Date: Fri, 12 Mar 2010 16:04:10 GMT Client-Peer: 208.68.72.130:80 Client-Response-Num: 1 X-AspNet-Version: 2.0.50727 X-Powered-By: ASP.NET new Object();r.error = new ajax_error('System.FormatException','Input +string was not in a correct format.\r\nCould not retreive parameters +from HTTP request.',0)new Object();r.error = new ajax_error('System.A +rgumentException','Object of type \'System.DBNull\' cannot be convert +ed to type \'System.Int32\'.',0)

Anyone have any ideas? If it matters, I'm on a Windows machine.

Replies are listed 'Best First'.
Re: Trying to Copy Live Http Replay
by Corion (Patriarch) on Mar 12, 2010 at 16:14 UTC

    At least, you're not sending the cookie the browser sends. I recommend comparing what your Perl script sends and what the browser sends, by use of Wireshark, or by using LWP::Debug.

      Using Wireshark, I found that \r\n is being converted to %5Cr%5Cn ... any thoughts how to stop that from happening?

        Use interpolating strings, or better, don't try to add \r\n yourself, as HTTP::Headers should format your headers anyway.

        1) you will definitelypossibly need the cookies. You can either manually add a cookie header with the value listed, or you can use a cookie jar.

        2) Try backing out from using the ua->post method to creating a post from HTTP::Request::Common so you can interact with and examine it.

        3) note that the outgoing request was text/plain. No URL encoding was done.

        Turns out you just need it in the format it wants, and will accept text/plain or x/www-form-encoded content.

        #!/usr/bin/perl use warnings; use strict; use HTTP::Request::Common; use LWP::UserAgent; use Data::Dumper; my %fields = ( "LeagueID" => "9\r\n", "GameDate" => "3-2-2010\r\n", "Season" => "2009-2010\r\n", "Refresh" => "false\r\n", "LastUpdateTime" => "01-01-1900\r\n", "type" => "Matchups\r\n", "RefreshStartTime" => "11-2-2010-1268372427240\r\n", "Week" => "\r\n", "conferenceID" => "", ); my $robot = LWP::UserAgent->new; use HTTP::Request::Common; my $url = 'http://scores.covers.com/ajax/SportsDirect.Controls.LiveScoresC +ontrols.Scoreboard,SportsDirect.Controls.LiveScoresControls.ashx?_met +hod=UpdateScoreboard&_session=no'; my $post = POST $url , \%fields; # $post would be url-encoded # 'content-type' => 'application/x-www-form-urlencoded', # print Dumper $post; my $content = qq{LeagueID=9\r\nGameDate=3-2-2010\r\nSeason=2009-2010\r +\nRefresh=true\r\nLastUpdateTime=01-01-1900\r\ntype=Matchups\r\nRefre +shStartTime=12-2-2010-1268408433293\r\nWeek=\r\nconferenceID=}; # your target url appears to accept either of these my $post2 = POST $url, Content => $content; #my $post2 = POST $url, 'content-type' => 'text/plain', Content => $co +ntent; print Dumper $post2; print Dumper $robot->request( $post2);
        at least I assume this is the sort of output you were expecting:
        '_content' => '{\'LeagueID\':9,\'League\':\'NBA\',\'GameDate\':\'3-2- +2010\',\'Season\':\'2009-2010\',\'Week\':\'\',\'GameResponseList\':[] +,\'LastUpdateTime\':\'01-01-1900\',\'RefreshStartTime\':\'12-2-2010-1 +268408433293\',\'Refresh\':true,\'PageType\':\'Matchups\',\'ScoreBoar +dHTML\':\'<div id="Scoreboard_9"><table class="scoreboard" border="0" +>\\r\\n <tr>\\r\\n <td class="scoreboard-left"><div + id="Game_9_781166" class="game-box"><span class="activetab" id="In_G +ame_Status_9_781166">Final</span><div class="soright" id="Title_9_781 +166">Boston at Detroit</div><span class="pick_button"><a href="http:/ +/contests.covers.com/sportscontests/makepicks.aspx?sportID=9&EventID= +781166"><img src="http://images.covers.com/scores/general/pick.gif" a +lign="absmiddle" alt="contest pick" border="0"
Re: Trying to Copy Live Http Replay
by BrowserUk (Patriarch) on Mar 12, 2010 at 16:12 UTC

    Total guess: Leave the \r\n off of your hash values.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      Nope. Neither does changing \r\n to \n