Tumbleweed Secure File Transfer and WWW::Mechanize

dhosek has asked for the wisdom of the Perl Monks concerning the following question:

I'm having difficulty with connecting to a Tumbleweed Secure File Transfer site. I can connect to the site using curl as follows: curl -v -A'Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16' https://ftp.polk.com But the following Perl code fails to connect:

#!/usr/bin/perl

use strict;
use warnings;
use lib 'lib';

use LWP::Debug qw(+);
use WWW::Mechanize;
use Data::Dumper;

my $polk_url      = 'https://ftp.polk.com';

my $mech = WWW::Mechanize->new(agent => 'Mozilla/5.0 (Macintosh; U; In
+tel Mac OS X; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16');
$mech->add_header(
    'User-Agent' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US;
+ rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16',
    'Accept'     => '*/*',
);

my $r = $mech->get($polk_url);

print $r->status_line;
print $r->content;
[download]

(I've left in some of the things I've tried to examine results/attempt to fix problems.) It appears that I'm not getting headers sent as I would expect from WWW::Mechanize. Any ideas?

Comment on Tumbleweed Secure File Transfer and WWW::Mechanize Select or Download Code

Replies are listed 'Best First'.
Re: Tumbleweed Secure File Transfer and WWW::Mechanize by oko1 (Deacon) on Aug 28, 2008 at 01:47 UTC
When I run the "curl" line and the script, I get exactly the same output, byte for byte (well, the script prepends a '200 OK', but otherwise identical): `Polk North America Secure File Transfer Login Name: [ ] Password: [ ] [Log In] Welcome to Secure File Transfer Polk North America Secure File Transfer is an enterprise file trans +fer system -- leading the industry in security, reliability, performanc +e, scalability, and extensibility.` [download] (I've elided some whitespace in the above.) It seems that your script works fine; if you're having problems, they most likely lie elsewhere. -- Human history becomes more and more a race between education and catastrophe. -- HG Wells	[reply] [d/l]
Re: Tumbleweed Secure File Transfer and WWW::Mechanize by Anonymous Monk on Aug 27, 2008 at 23:35 UTC
Here's what I get D:\>lwp-request -USsexd https://ftp.polk.com LWP::UserAgent::new: () LWP::UserAgent::request: () LWP::UserAgent::send_request: GET https://ftp.polk.com LWP::UserAgent::_need_proxy: Not proxied LWP::Protocol::http::request: () LWP::Protocol::collect: read 397 bytes LWP::UserAgent::request: Simple response: Unauthorized Enter username for FileDriveWWW at ftp.polk.com:443: GET https://ftp.polk.com User-Agent: lwp-request/5.810 GET https://ftp.polk.com --> 401 Authorization Required Connection: close Date: Wed, 27 Aug 2008 23:34:05 GMT Server: SecureTransport/4.7 WWW-Authenticate: Basic realm="FileDriveWWW" Content-Type: text/html; charset=UTF-8 Client-Date: Wed, 27 Aug 2008 23:38:55 GMT Client-Peer: 63.87.34.12:443 Client-Response-Num: 1 Client-SSL-Cert-Issuer: /O=VeriSign Trust Network/OU=VeriSign, Inc./OU +=VeriSign International Server CA - Class 3/OU=www .verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign Client-SSL-Cert-Subject: /C=US/ST=Michigan/L=Southfield/O=R. L. Polk & + Co./OU=Terms of use at www.verisign.com/rpa (c)00 /CN=ftp.polk.com Client-SSL-Cipher: AES256-SHA Client-SSL-Warning: Peer certificate not verified Client-Transfer-Encoding: chunked Title: 401 Authorization Required [download] Its asking for username/password, and I have none.	[reply] [d/l]
Re^2: Tumbleweed Secure File Transfer and WWW::Mechanize by dhosek (Beadle) on Aug 28, 2008 at 00:50 UTC
It gives that result if you provide no user agent. curl will give the same result without the -A parameter that I gave. Interestingly, I'm getting a 500 response rather than a 401 response from the same command on my system.	[reply]
Re^3: Tumbleweed Secure File Transfer and WWW::Mechanize by Anonymous Monk on Aug 28, 2008 at 01:28 UTC
It gives that result if you provide no user agent. curl will give the same result without the -A parameter that I gave. Interestingly, I'm getting a 500 response rather than a 401 response from the same command on my system. Show the debug data, you know the headers and stuff	[reply]