in reply to Re^3: blocking site scrapers
in thread blocking site scrapers

What's wrong with dynamically altering firewall rules? Before answering you should perhaps consider that firewalls can be used for tarpitting (i.e. slowing down connections to the point of unusability) or rate-limiting individual addresses or address ranges, as well as simple blocking. In fact, if you have to resort to an IP-based policy (generally a bad idea), a well-implemented firewall solution is usually a better idea than server-side request mangling.

To answer the OP's question, if you're on Linux you may want to look at the "recent" iptables extension. This article provides an introduction on how to use it. If you're on a different OS, have a look at that OS's firewall documentation.


All dogma is stupid.

Replies are listed 'Best First'.
Re^5: blocking site scrapers
by chargrill (Parson) on Feb 07, 2006 at 15:48 UTC

    /me humbly searches through his own httpd.conf and finds

    SetEnvIf Request_URI "winnt/system32/cmd\.exe" worm # etc ... CustomLog "|exec sh" "/sbin/route -nq add -host %a 127.0.0.1 -blackhol +e" env=worm

    ... so I guess to answer your question, the answer is that nothing is wrong with it per se. This was a somewhat popular method to block nimda, code red, sadmind, etc from doing too much damage to web servers a few years ago. More can be read here: log monitors and here: securityfocus. These links even suggest that indeed local or upstream firewalling would be more efficient.



    --chargrill
    $/ = q#(\w)# ; sub sig { print scalar reverse join ' ', @_ } + sig map { s$\$/\$/$\$2\$1$g && $_ } split( ' ', ",erckha rlPe erthnoa stJu +" );