in reply to Re^2: perl regex or module that identifies bots/crawlers
in thread perl regex or module that identifies bots/crawlers

The company I work for gets over 9 million hits per day. Many are from bots but bots typically just act like regular users in order to crawl your site for caching/search/things like that.

I would take some time to look at your code to see what is going on. 25K visitors causing a load of over 200 points to something other then just spiders crawling your site. What would happen if you had 25K valid users? You're site crashes? That's probably not what you want.

Where I work I use a combination of ip address and user agent string to identify search engines. We don't block them per say(we still want to show up in google searches) we just don't give them shopping carts or do other tracking type work with them

D
  • Comment on Re^3: perl regex or module that identifies bots/crawlers