in reply to CGI to Pull links off webpage fails on second run

It would probably be a good idea to check what data you are getting back. To test, take out the link extractor and just print the raw HTML you get back from the request. I'd try that for both useragents (the default and the MSIE one below) and see what gets printed.

Don't know if this is any indication of you being blocked, but using your code you posted, I can run the script successfully at least 5 or 6 times (that's all I tried) in a row.

  • Comment on Re: CGI to Pull links off webpage fails on second run