Re: Re: Thwarting Screen Scrapers

Thanks...this is the kind of input I was looking for.

Obviously any type of measure has a countermeasure, and if it works on a browser, It would work in LWP ( or some other interface ).

The addition of a timestamp into the hash calculation is an interesting one.

We've already worked out a method of using dynamically generated form field names from a hash of the session key. Adding the timestamp purturbs it a bit, and keeps someone from keeping a session alive over a long period of time.

dws++...thanks again.

Comment on Re: Re: Thwarting Screen Scrapers

Replies are listed 'Best First'.
Re: Re: Re: Thwarting Screen Scrapers by dws (Chancellor) on Jul 18, 2002 at 17:40 UTC
The addition of a timestamp into the hash calculation is an interesting one. Interesting, but not what I intended to suggest. Using a timestamp when generating the hash needlessly complicates verification. What I meant to suggest was that you save a timestamp when you record generated IDs. This gives you an easy way to "time out" forms, and flush abandoned forms out of your back-end database. It also sets you up for doing some analysis on things like average submit time (the gap between your generating the form, and a user submitting it). A really low submit time is an indication that there's a bot on the other end of the line.	[reply]

Replies are listed 'Best First'.

Re: Re: Re: Thwarting Screen Scrapers
by dws (Chancellor) on Jul 18, 2002 at 17:40 UTC

The addition of a timestamp into the hash calculation is an interesting one.

Interesting, but not what I intended to suggest. Using a timestamp when generating the hash needlessly complicates verification.

What I meant to suggest was that you save a timestamp when you record generated IDs. This gives you an easy way to "time out" forms, and flush abandoned forms out of your back-end database. It also sets you up for doing some analysis on things like average submit time (the gap between your generating the form, and a user submitting it). A really low submit time is an indication that there's a bot on the other end of the line.

[reply]