Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re: Using LWP (or some other module) to Dowload HTML with Cookie Session ID

by Corion (Patriarch)
on Nov 28, 2022 at 07:36 UTC ( #11148421=note: print w/replies, xml ) Need Help??


in reply to Using LWP (or some other module) to Dowload HTML with Cookie Session ID

An easy way to get a start is by converting your Curl command to a Perl program using (my) Curl to Perl converter:

#!perl use strict; use warnings; use LWP::UserAgent; my $ua = LWP::UserAgent->new( 'send_te' => '0' ); my $r = HTTP::Request->new( 'GET' => 'https://example.com/', [ 'Accept' => '*/*', 'User-Agent' => 'curl/7.55.1', 'Cookie' => 'session=536...035a' ], ); my $res = $ua->request( $r, ':content_file' => 'HTML_output.txt' ); __END__ Created from curl command line curl https://example.com --cookie "session=536...035a" -o "HTML_output +.txt"

I'm not sure if your Firefox stores its cookies in a cookies.txt file still or if it stores them in an SQLite database nowadays.

Replies are listed 'Best First'.
Re^2: Using LWP (or some other module) to Dowload HTML with Cookie Session ID
by Leudwinus (Beadle) on Nov 28, 2022 at 23:50 UTC

    Thank you, Corion for that example and link to your page which I have bookmarked!

    It looks like the content I need is then stored in $res->{_content}. I had to first install LWP::UserAgent and LWP::Protocol::https to get this to work though.

    I guess I need to get more familiar with the LWP, LWP::UserAgent and HTTP::Request modules.

    What is the difference between LWP::UserAgent and Mojo::UserAgent, the latter which was also suggested to me to try to solve this?

    Thanks again!

      No - please use $res->decoded_content instead of reaching into the HTTP::Response hash.

      The difference between LWP::UserAgent and Mojo::UserAgent is mostly that LWP::UserAgent doesn't allow for parallel requests. Mojo::UserAgent is a bit more complex due to allowing for parallelism, but it also offers some more convenience stuff like handling JSON replies directly.

        Ok, thank you again Corion! As you can clearly tell, I have no idea what I am doing!

        And thank you for clarifying the differences between LWP::UserAgent and Mojo::UserAgent.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11148421]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (4)
As of 2023-01-30 05:32 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?