I've tried disabling cookies in Opera and Lynx, and I managed to basically simulate whats happening in my program. I have now set the search to 10 results per page, therefore I now have to traverse 11 pages in order to read all the results. With both my program, and the browsers ( with cookies disabled) the serch fails at about page 3 or 4. Therefore, I'm guessing now that my cookies are not working as I expect. | [reply] |
Ok, I think I might have solved it. Perlmonk.org is down momentarily so I cant confirm as of yet.
Before, if I had 10 pages of results to retrieve, I simple GoT each one using a for loop, which updated the search string for each GET. If I failed to get the 3rd page, then pages 4, 5, 6, 7 etc all failed too. But, if I was using my browser, I could skip a page of the search and things continued to work fine. So say I was on page 5, instead of hitting next, I could change the search string and check page 7. That led me to think that my prog was failing because my session id suddenly became out of date and it would therefore make sense that nothing else after that would work either, which is what was happening.
But now, I simply recall a page when it fails, and everything suddenly works again and I can then get 4, 5, 6, 7 etc. :) - once I've sucessfully received page 3
Why does a page fail at all? - well I ran the loop above in such a way that it would continue infinetly - say 0, 1, 2, 3, 0, 1, 2, 3...
I noticed that page 2 or 3 would normally fail at the fist attempt, but once I got passed that, pages 2,3,0,1,2,3,0,1,2,3... all worked fine - it only seems to fail once.
So now I simply decrement my loop counter on a fail, and set an error watch to make sure I dont get caught in this loop indefinitely
Anyway, thanks to all those who gave suggestions ;)
| [reply] |
Perlmonk is up again - and it does indeed look like its working. Infact, the script seems to have become more successful since I last tried it, in the sense that in most cases, it now completes the search with out failing once.
| [reply] |