Trying to screen scrape with WWW::Scripter

redmage has asked for the wisdom of the Perl Monks concerning the following question:

Hello all, I wrote a program with WWW::Scripter that did a screen scrape of a particular web site. This site has now completely changed their format and have moved to https from http. I've tried to figure out how to re-screenscrape them but so far I've had no luck. My real problem is that, since the data is encrypted, I can't figure out how to compare what I'm sending as POST information compared to what the browser is sending so that I can modify my code accordingly. How can I figure out how to match what the browser is sending so that I can put that in my perl code? Thanks -- redmage

Comment on Trying to screen scrape with WWW::Scripter

Replies are listed 'Best First'.
Re: Trying to screen scrape with WWW::Scripter by nikosv (Deacon) on Oct 19, 2015 at 06:29 UTC
You need a proxy that sits in between to capture and decrypt the traffic. One such proxy is Fiddler,see: Decrypting HTTPS-protected traffic	[reply]
Re^2: Trying to screen scrape with WWW::Scripter by redmage (Initiate) on Oct 19, 2015 at 07:31 UTC
I believe that Fiddler is windows-only. Is there a linux based solution?	[reply]
Re^3: Trying to screen scrape with WWW::Scripter by nikosv (Deacon) on Oct 19, 2015 at 07:38 UTC
Burp proxy	[reply]
Re: Trying to screen scrape with WWW::Scripter by Corion (Patriarch) on Oct 19, 2015 at 06:59 UTC
I think the Mozilla Live HTTP Headers don't care about whether the target site uses http or https.	[reply]