No matter what you do implementation-wise, I don't see too many options as to what you can do about this. You have the data available to the public, so how are you going to determine who's "real" and who's a program trying to get all your results?

That would be like perlmonks creating a security layer that only allows a certain computer to access a set number of nodes per hour. Things just don't work like that. As far as going by IP address, I wouldn't even consider that. Too many proxies, dynamic IPs and such going around. Using an IP address to pinpoint a specific user just doesn't work as well as it might have in the past.

The only method I can think of that would be scalable and programatically simple would be to require authentication before being permitted to browse through the listings. Then, you could set limits on number of listings viewable per week or something like that.

But even that is circumventable. The person harvesting your database simply registers multiple usernames (as many as it takes) to do the job (first user grabs first 30 listings, second one grabs next 30, etc etc). So there is no real way that I can think of to 100% effectively solve this "problem". The data is freely available. If your site dishes it out, there's no way to stop it from being programatically stolen.

In reply to Re: State-of-the-art in Harvester Blocking by Coruscate
in thread State-of-the-art in Harvester Blocking by sgifford

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.