in reply to Subdir globs

You might have a look at LWP::RobotUA. It's a really good place to start if you want to build a robot. That follows the recommendation of Ozymandius.