You might want to use LWP::RobotUA. It obeys the robots.txt file (if present), allows you to control the number of requests per minute, and has many other polite features.