Hi--

I am trying to grab a page with the following code: 
use LWP::Simple;
use LWP::UserAgent;

$browser = LWP::UserAgent->new;

$browser->default_headers->push_header('User-Agent' => 'Mozilla/4.0 (c
+ompatible; MSIE 7.0; Windows NT 5.1; iOpus-I-M; GTB6; .NET CLR 2.0.50
+727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; InfoPath.2; .NET 
+CLR 3.0.4506.2152; .NET CLR 3.5.30729)');
$browser->default_headers->push_header('Host' => "brtweb.phila.gov");
$browser->default_headers->push_header('Connection' => "close");

$response = $browser->get('http://brtweb.phila.gov/brt.apps/Search/Sea
+rchResults.aspx?id=6546003202');
print $response->content;

[download]
Nothing fancy or anything.

But yet when I pull down the page with the above code, the results I get most, but not all, of the same page if I use a browser to do the job. Principally, I am missing 2 chunks that are part of a 3-part tabbed view on the page (in this case, Perl only gets "Account Information", but not the "Account Details" or "Property Valuation" tabs. They never show up from a Perl request. And I do not see another sub-request being made to pull the data down.

I have tried sending every header needed, as shown by a trace from www.rexswain.com (Thanks!) and even that site is able to fetch the entire page.

Any ideas as to what I am not providing/doing wrong here?

Thanks!

In reply to lwp not retieving the same page as from a browser by zuma53

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.