in reply to Testing a web crawler
I ended up hacking something together using HTTP::Daemon. This worked, though I think there should be a more turn key approach.
Thanks for the suggestions