I had this problem a while back and as I recall, it did in fact have to do with redirects. I am too ashamed to post any of my lame code, but see How to tell if a URL returned a Location: header? for some useful info.
Comment on Re: Cannot retrieve HTML for some pages with LWP