Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

Re: Cookies, JavaScript and User Agent Problems

by keszler (Priest)
on Oct 25, 2011 at 01:01 UTC ( [id://933512]=note: print w/replies, xml ) Need Help??


in reply to Cookies, JavaScript and User Agent Problems

Your first, commented out, cookie_jar line is fine; it should be outside the loop. I'd put it right after you create $ua. What seems to me to be missing is the bit where you log in. It's likely that the site only assigns session cookies, so you need to login each time you start scraping.

  • Comment on Re: Cookies, JavaScript and User Agent Problems

Replies are listed 'Best First'.
Re^2: Cookies, JavaScript and User Agent Problems
by koolgirl (Hermit) on Oct 25, 2011 at 01:10 UTC

    Thank you. I am trying to create that, but see - there's either a log in with name/password, or a press this button to log in publicly, so that kind of throws a wrench in the mix....

      ** Sorry, I wrote this before I saw that you already mentioned WWW-Mechanize

      If the login is a problem, then you could simply use WWW-Mechanize. Generally speaking you can usually figure a way with WWW-Mechanize to get through a login screen, after a bit of playing around ... And you can easily retrieve data on any subsequent pages then.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://933512]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (5)
As of 2024-04-16 13:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found