advait,
I am suprised (and happy) to see no one mentioned checking the HTTP referer. It, like every other technique to reduce the amount of undesired traffic, is not a complete solution. That is the nature of the beast.

What I am surprised is that no one mentioned defining a Terms Of Service agreement with folks using the site (though robots.txt is a good start). Explain in exactly what ways it is acceptable to interface (API, how frequent are requests, etc). This tells people what you expect and what you will do if they don't abide by those rules.

Then all you can do is protect yourself, identifying abusers, and take other steps to block them.