The company I work for gets over 9 million hits per day. Many are from bots but bots typically just act like regular users in order to crawl your site for caching/search/things like that.

I would take some time to look at your code to see what is going on. 25K visitors causing a load of over 200 points to something other then just spiders crawling your site. What would happen if you had 25K valid users? You're site crashes? That's probably not what you want.

Where I work I use a combination of ip address and user agent string to identify search engines. We don't block them per say(we still want to show up in google searches) we just don't give them shopping carts or do other tracking type work with them

D

In reply to Re^3: perl regex or module that identifies bots/crawlers by Sartan
in thread perl regex or module that identifies bots/crawlers by argv

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.