Re: State-of-the-art in Harvester Blocking

i thought the first question that needs to be answered is "why block?" if you simply don't want competitors to easily suck in addresses and prices then either don't list prices or put the prices as gifs, then at least it will take manual labor to view the pages and write down the prices. You could potentially be also restricting legitimate (computer savvy) buyers, who do their homework. A fairly hard to work around page would consist of a session, and the session being modified slightly each time a user comes to your page...and the session must follow the proper path to succeed. this however, would be a pain to frequent users of your site, as things would look different each time they try to do the same thing (get some listings) Perhaps you alarmed the non-tech people in your company too much when you discovered harvesters in your logs. i certainly would not worry about it unless bandwitdth was being adveresely affected.

Comment on Re: State-of-the-art in Harvester Blocking

Replies are listed 'Best First'.
Re: Re: State-of-the-art in Harvester Blocking by sgifford (Prior) on Nov 23, 2003 at 18:10 UTC
Yeah, I'm not sure why the data needs to be blocked---I was just hired to do the coding, not set policy. Another reason for my concern is that we've looked at incorporating other databases, and to do that we have to agree to do something about harvesting: An IDXP displaying the IDX Database or any portion thereof shall make reasonable efforts to avoid "scraping" of the data by third parties or displaying of that data on any other Web site. Reasonable efforts shall include but not be limited to: Monitoring the Web site for signs that a third party is "scraping" data and Prominently posting notice that "Any use of search facilities of data on the site, other than by a consumer looking to purchase real estate, is prohibited." This section places a burden on the Broker and the Broker s Web site host to monitor their site. If it appears that a large number of hits is coming from a particular domain on the Web and that these hits may be the result of an automated process designed to gather or scrape data from the Broker's Web site for use somewhere else for a commercial purpose, the Broker must notify (Agency). So, I'd have to do a lot of convincing before I'd be able to say, "Guys, just don't worry about it!". And yes, I realize I could do just the two things above and probably we'd be safe contractually, but if I agree to make an effort to stop harvesters, I'd like to make sure I'm doing my honest best.	[reply]

Replies are listed 'Best First'.

Re: Re: State-of-the-art in Harvester Blocking
by sgifford (Prior) on Nov 23, 2003 at 18:10 UTC

Yeah, I'm not sure why the data needs to be blocked---I was just hired to do the coding, not set policy.

Another reason for my concern is that we've looked at incorporating other databases, and to do that we have to agree to do something about harvesting:

An IDXP displaying the IDX Database or any portion thereof shall make reasonable efforts to avoid "scraping" of the data by third parties or displaying of that data on any other Web site. Reasonable efforts shall include but not be limited to:

Monitoring the Web site for signs that a third party is "scraping" data and
Prominently posting notice that "Any use of search facilities of data on the site, other than by a consumer looking to purchase real estate, is prohibited."
This section places a burden on the Broker and the Broker s Web site host to monitor their site. If it appears that a large number of hits is coming from a particular domain on the Web and that these hits may be the result of an automated process designed to gather or scrape data from the Broker's Web site for use somewhere else for a commercial purpose, the Broker must notify (Agency).

So, I'd have to do a lot of convincing before I'd be able to say, "Guys, just don't worry about it!".

And yes, I realize I could do just the two things above and probably we'd be safe contractually, but if I agree to make an effort to stop harvesters, I'd like to make sure I'm doing my honest best.

[reply]