in reply to obeying robot rules

Yes. Use WWW::Robot for what it was designed to do.

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

In Section Seekers of Perl Wisdom