One of the things to look for when creating a spider or robot is patterns. For instance, if you want to submit a form, and the form uses the same name and values for each field and form name, it is very easy to create a spider to submit a form. If the form names vary, you have to look for sections of the HTML which are similar. If there is variation in the HTML which does not alter the lok of the page, things get very hard very quickly. Another thing that makes spidering very hard is if a lot of Javascript is used, because there are no modules to create a web page based on what the javascript says to do.

These are some of the things that made my life very hard when a client asked for spiders which helped create a site that was a clone of AddAll.com. I would suggest using them to your advantage. Be devious. Use javascript like document.location = "http://www.newwebpage.com" to change locations in ways that a spider will have trouble keeping up with. Alter form names and the names of the input so that they contain random charachters (you can keep track of their real values in a database. Use a random key you can get from a cookie you send the user to look up what the real field names are)

Chances are, if you do all these things, people will leave your site alone. Now, granted, many programmers / hackers -- given enough time and energy -- can overcome these problems. But by frustrating your attackers it is likely they will look for an easier target to pick on.

Hope that helps,

Vautrin