Re: Cookies, JavaScript and User Agent Problems

Your first, commented out, cookie_jar line is fine; it should be outside the loop. I'd put it right after you create $ua. What seems to me to be missing is the bit where you log in. It's likely that the site only assigns session cookies, so you need to login each time you start scraping.

Comment on Re: Cookies, JavaScript and User Agent Problems

Replies are listed 'Best First'.
Re^2: Cookies, JavaScript and User Agent Problems by koolgirl (Hermit) on Oct 25, 2011 at 01:10 UTC
Thank you. I am trying to create that, but see - there's either a log in with name/password, or a press this button to log in publicly, so that kind of throws a wrench in the mix....	[reply]
Re^3: Cookies, JavaScript and User Agent Problems by DanielSpaniel (Scribe) on Oct 25, 2011 at 02:12 UTC
** Sorry, I wrote this before I saw that you already mentioned WWW-Mechanize If the login is a problem, then you could simply use WWW-Mechanize. Generally speaking you can usually figure a way with WWW-Mechanize to get through a login screen, after a bit of playing around ... And you can easily retrieve data on any subsequent pages then.	[reply]