Use LWP::UserAgent for more fine grained control. You can also fake being IE (agent) rather than LWP which makes it just that bit harder to catch you (see below). If you can't bypass the proxy one reasonably reliable trick to make proxies refresh is to add something to the query string each time either ?random_stuff=12345 or &random_stuff=12345 depending if there is already a query_string. The proxy sees this as a new URL so goes to fetch it, the target server will *usually* ignore the extra data (although some scripts *will* implode deliberately or accidentally). As always YMMV but this seems the most reliable method of making sure you have fresh data.

If you are trying to rip data from most live feed stock price websites they will almost certainly block you at some stage. You are neither the first nor the last to want something for nothing. There are plenty of relatively low cost XML data feeds about. If you are parsing the data out it generally breaks when the web designers update the site. 

use LWP::UserAgent;
use Data::Dumper;

my $domain = 'http://blah.com';

my $ua = LWP::UserAgent->new;
$ua->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Reading
+ Logs - You must be bored ;-)');

# use proxy....
my $proxy   = 'http://some.proxy.au:8080';
$ua->proxy( 'http', $proxy );

# or don't use proxy......
$ua->no_proxy($domain,...)

my $request = HTTP::Request->new( 'GET', $URL );
my $response = $ua->request( $request );
print Dumper $response;

[download]

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

In reply to Re: How to Get around Proxy by tachyon
in thread How to Get around Proxy by Kanishka

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.