in reply to Site Crawler

Just to be more complete: I have tried $contents = get($url); instead of all of the LWP::UserAgent, HTTP::Request, etc stuff. It didn't work either.