Maybe I'm shooting in the wrong direction, but isn't a redirect loop just that? One website issues a redirect to another website, which in turn redirects back to the first website and you're stuck in an endless loop with no hope of reaching either website.
I once caused this on my own website using mod_rewrite to add a '/' to URLs. I mistakenly issued a re-direct with the modified URL instead of just passing through to serve up the page. Firefox got the redirect and carried on with the modified URL ending in '/'. IE got the redirect, stripped the trailing '/' and tried again, and again, and again. It was a very effective IE blocker.
But I guess my point is, I wouldn't be surprised if you came across a mis-configured (intentionally?) web server that's causing a redirect loop so this error must be expected. It may also be that LWP::UserAgent is modifying the URL in some way that keeps the redirect looping. | [reply] |
The thing is, I can do the redirection manually. I can find the url redirection link in the retrieved web page, and use it to retrieve the web page I want. This works, but it is very long and ugly, and I would rather use the LWP::UserAgent redirection if I can.
Also, the page that I get back with the redirection loop is a page that can be redirected (it has the redirection url link), so I don't think that something is wrong with the website.
| [reply] |
Do you have a page example?
According to the docks the redirect limit in LWP::UserAgent is set to 7 unless you change it with the 'max_redirect' option. I can't think of a reason why you would need to set it beyond 7. If you've some how set it to 0 then that would explain a few things.
| [reply] |
just keep track of the pages visited and if there is a self-link from the current page,don't
follow it(you can use a hash to keep track of pages visited)
| [reply] |