Robots.txt files only appear on a minority of sites, but should always be respected. A good first step is to start with LWP::RobotUA (or LWP::Parallel::RobotUA), which will implement the Robots Exclusion Standard for you.