While robots.txt is an established standard for regulating the behavior of robots, the non-existence of a robots.txt is not a license to violate copyright. Many sites would want robots from Google and other search engines to index their pages, but they don't want any random person scraping their content and putting it up on another site. You could make a case for downloading content for personal use, but there's definitey gray areas out there.

The moral of the story is that the legality definitely depends on the use of downloaded information.