in reply to Re: Re: Automating a search using perl
in thread Automating a search using perl
More generally, sites can analyze the "signature" of requests to identify if they want to block you or not. By signature i mean anything that can make your requests stand out from those of the other 99% of their traffic. They might do it based on your User-Agent, or some other HTTP header that is unique to the API you are using, or they might do it based on some combination of things that help identify people who are being decieptful (if your User-Agent says you're Netscape 6, but you use "HTTP/1.0", that's a dead give away ... other more subtle things might be descrepencies in what HTTP headers you send vs. the headers that Netscape 6 ALLWAYS sends.
Bottom line: play nice. If you get blocked, you probably deserved it.
|
|---|