I've found that some really rotten bots provide user agent strings claiming they're googlebot or slurp, but their IP addresses are nowhere near either of those two companies. The ones I've found being most abusive don't seem to move around much in IP space, so I just drop their packets via iptables and let it go at that. Because I have some traps set up on my web site, I check my server logs regularly to see who's wandered into globally excluded directories, and every agent claiming to be googlebot or slurp who's gone there has been an imposter.

I do reject a few user agent strings, though. "PHP Script", "MSRBOT", and "Java*" are denied via (my) Apache's configuration because they're commonly found attempting to abuse my web-to-mail gateway.

To answer the question you asked: I'm not aware of a ready-made module that will do what you want done.

In reply to Re: perl regex or module that identifies bots/crawlers by gloryhack
in thread perl regex or module that identifies bots/crawlers by argv

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.