in reply to Cookies, JavaScript and User Agent Problems
IMHO, you need to see what's actually being sent between an actual browser and the site you're scraping. I use:
LiveHTTPHeaders Wireshark
to find out this kind of information, then code accordingly.
HTH,