I can reproduce that it's not working:

Unquoted string "http" may clash with future reserved word at tmp.pl l
+ine 2.

[download]

This is most likely because you are missing a starting single quote in line 2.

Maybe you want to elaborate further in what sense it is "not working".

In the end, no server can discern whether it's a browser+human or a well-crafted Perl script that sends the requests over the wire, so your goal is to mimic what the combination of browser+human sends over the wire with your Perl script. To do that, you need tools to compare what your browser sends against what your Perl script sends.

Here's my checklist of things to do while trying to scrape a website:

  1. Does the site work when manually browsing to it using a browser?
  2. Does the site use frames? If so, does the target frame page work when manually browsing to it using a browser?
  3. Does the site work when manually browsing to it with JavaScript disabled?
  4. What do the Firefox HTTP Live Headers output?
  5. Is the output of the HTTP Live Headers identical to what the wireshark tcp dump of your script says?

My current web scraping tools are WWW::Mechanize for navigation (with WWW::Mechanize::Shell for quick exploration) and Web::Scraper for data extraction.

Update: You changed your post to include a bit more of your code, but as you still don't tell us where and how it fails for you, and as it still contains the typo, that is of no more help than the previous content.

In reply to Re: Help with LWP GET request by Corion
in thread Help with LWP GET request by mhearse

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.