How can I use LWP to log into digest-auth site and do post() (converting from PHP CURL)

fgsfds100 has asked for the wisdom of the Perl Monks concerning the following question:

Hi. Sorry if this type of thing has been asked before. I looked at a bunch of other topics here and on other sites but nothing has helped.

I'm trying to convert a PHP class into perl. It uses CURL to log into (with digest authentication) and post some parameters in order to collect trophy data from the playstation network. Original PHP code can be found here if you're curious: http://psnapi.org/?page=forum&forum=4&topic=69

My problem right off the bat is, I can't even get it to log in properly using perl. I just keep getting error 401. If I try to log in with a browser, it works (HTTP 200 OK) but then I get a 404 error because I haven't posted anything. Also, if I try to log into my own site (basic_auth), it DOES work.

I'd like to stick with using LWP and HTTP if at all possible. I know there are CURL and WWW::Mechanize modules for perl, but every version I've tried to install seems to be incompatible with my version of perl.

This is my code at the moment. I've also tried using basic_auth, get(), head() and post(), all with different parameters but everything just gives me error 401. Also, for the credentials method, I'm unsure which port the site uses. Could that be my whole problem and, if so, how can I find the necessary port?

#!C:/Perl/bin/perl.exe
use LWP;

$site[0] = "http://searchjid.usa.np.community.playstation.net/basic_vi
+ew/func/search_jid";
$site[1] = "http://searchjid.eu.np.community.playstation.net/basic_vie
+w/func/search_jid";
$site[2] = "http://searchjid.jpn.np.community.playstation.net/basic_vi
+ew/func/search_jid";
$user = "c7y-basic01";
$pass = "A9QTbosh0W0D^{7467l-n_>2Y%JG^v>o";

for ($i=0; $i<@site; $i++)
 {
  $browser = LWP::UserAgent -> new;
  $browser -> agent("PS3Community-agent/1.0.0 libhttp/1.0.0");
  $browser -> credentials("searchjid.usa.np.community.playstation.net/
+basic_view/func/search_jid:80","c7y-basic",$user=>$pass) . "\n\n";
  print $browser -> head($site[i]) -> as_string . "\n\n";

  # ------ alternate method ------
  # $browser = LWP::UserAgent -> new; 
  # $browser -> agent("PS3Community-agent/1.0.0 libhttp/1.0.0"); 
  # $req = HTTP::Request -> new(GET => $site[$i]); 
  # $req -> authorization_basic($user,$pass); 
  # print $browser -> request($req) -> as_string . "\n\n"; 
  # ------------------------------
 }
[download]

---- And this is the output I always get ---

HTTP/1.1 401 Authorization Required
Connection: close
Date: Sat, 21 Apr 2012 03:54:44 GMT
WWW-Authenticate: Digest realm="c7y-basic", nonce="XXXXXXX+BAA=XXXXXXX
+XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", algorithm=MD5, qop="auth"
Content-Type: text/html; charset=iso-8859-1
Client-Date: Sat, 21 Apr 2012 03:54:47 GMT
Client-Peer: 198.107.128.164:80
Client-Response-Num: 1
[download]

What am I doing wrong? (The XXXXX stuff is random every time).

Comment on How can I use LWP to log into digest-auth site and do post() (converting from PHP CURL) Select or Download Code

Replies are listed 'Best First'.
Re: How can I use LWP to log into digest-auth site and do post() (converting from PHP CURL) by zwon (Abbot) on Apr 21, 2012 at 05:39 UTC
What am I doing wrong? Except for posting a password on a public website? ;) You aren't using strict and warnings. You are using C style for loop (not really wrong, but not very perlish). You're adding `"\n\n"` to the result returned by the `credentials` method, which is useless. And the first argument of the `credentials` is wrong. Here's a working example: use 5.010; use strict; use warnings; use LWP::UserAgent; my @sites = ( "searchjid.usa.np.community.playstation.net", "searchjid.eu.np.community.playstation.net", "searchjid.jpn.np.community.playstation.net", ); my $path = "/basic_view/func/search_jid"; my $user = "c7y-basic01"; my $pass = "A9QTbosh0W0D^{7467l-n_>2Y%JG^v>o"; for (@sites) { my $browser = LWP::UserAgent->new; $browser->agent("PS3Community-agent/1.0.0 libhttp/1.0.0"); $browser->credentials( "$_:80", "c7y-basic", $user => $pass ); say $browser->head( "http://" . $_ . $path )->as_string; } [download]	[reply] [d/l] [select]
Re^2: How can I use LWP to log into digest-auth site and do post() (converting from PHP CURL) by fgsfds100 (Initiate) on Apr 22, 2012 at 12:52 UTC
Thank you so much, this worked perfectly! I don't think I ever would have got this working by myself, but I'm glad I was fairly close. And yes, I realize posting that password wasn't a great idea but I felt that, apart from being necessary to truly see if the program could connect to that site specifically, it doesn't grant access to anything significant and it was already included in the PHP code on the site I linked to. Also thanks for the tip on the for loop.	[reply]