Unfortunately. I had to tighten the screws on my private server as well. Most of those scrapers are really, really, dumb, too. When encountering a public repository (both git and mercurial), instead of just pulling the repo (a rather efficient operation), they just follow through the web pages and generate every page in every which way. Still working on some smarter rules, but so far i managed to reduce traffic to my server by (very roughly) 90% without affecting most legitimate users.

There are still a few things i want to implement to detect bot activity even better and have to ability to automatically block specific subnets when bot activity is detected from those IP's. But that's all very specific to my private server and unfortunately wont be applicable to the monastery.

PerlMonks XP is useless? Not anymore: XPD - Do more with your PerlMonks XP
Also check out my sisters artwork and my weekly webcomics

In reply to Re^3: Unable to connect by cavac
in thread Unable to connect by choroba

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.