LWP oddity

adamsj has asked for the wisdom of the Perl Monks concerning the following question:

So we're using LWP to pull data off a server by pretending we're a browser and going through a proxy--or we were, until firewall issues cut us off. They've added us to the allow list and should be getting through, but we're not.

"Okay," I says to myself, "if we can't go through the proxy 'cause it thinks we're it instead of us, I can just take the proxy setting out, and the request will just whiz right through."

Instead, now the script dies while I'm stepping through it in the debugger. This bothers me.

my $list_request = HTTP::Request->new(GET => $source_dir);
my $agent = LWP::UserAgent->new;
$agent->protocols_allowed(['ftp', 'http']);
#$agent->proxy('ftp' => 'http://www.nunya.biz:8080');
my $list_response = $agent->request($list_request);
my @lines = grep (s!^<A HREF="^\w+($loaddate)($file_pattern)([\w.]*).*
+$!$2$3!, (split(/\n/,$list_response->content)) );
[download]

Now, with the proxy setting taken out, it fails in the debugger on the next line.

I'm puzzled--any help?

adamsj

They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen

Comment on LWP oddity Download Code

Replies are listed 'Best First'.
Re: LWP oddity by trs80 (Priest) on Nov 13, 2002 at 00:40 UTC
You can turn on LWP debugging: `require LWP::Debug; LWP::Debug::level('+');` [download] if you haven't already. Update: Forgot the require LWP::Debug the first time	[reply] [d/l]
Following up: Re: LWP oddity by adamsj (Hermit) on Nov 13, 2002 at 18:12 UTC
I hadn't thought to set LWP::Debug (and did you know that Perl and LWP doesn't deal with LWP::Debug at all?), but now I have, and am getting much the same result when the proxy statement is removed. Without proxy, the output from LWP::Debug is: `LWP::UserAgent::request: () LWP::UserAgent::send_request: GET ftp://user:password@10.10.10.10/outg +oing/ LWP::UserAgent::_need_proxy: Not proxied LWP::UserAgent::send_request: ftp URLs are among LWP::UserAgent=HASH(0 +x83ae480)'s allowed protocols (ftp http) LWP::MediaTypes::read_media_types: Reading media types from /my/lib/pe +rl5/site_perl/5.6.1/LWP/media.types LWP::Protocol::ftp::request: () LWP::Protocol::MyFTP::new: ()` [download] and here the debugger dies, along with the script. With the proxy statement left in, I get: LWP::UserAgent::request: () LWP::UserAgent::send_request: GET ftp://user:password@10.10.10.10/outg +oing/ LWP::UserAgent::_need_proxy: Proxied to http://www.nunya.biz:8080 LWP::UserAgent::send_request: http URLs are among LWP::UserAgent=HASH( +0x83d28b0)'s allowed protocols (ftp http) LWP::Protocol::http::request: () LWP::Protocol::collect: read 315 bytes LWP::Protocol::collect: read 4096 bytes LWP::Protocol::collect: read 4096 bytes LWP::Protocol::collect: read 2954 bytes LWP::Protocol::collect: read 1460 bytes LWP::Protocol::collect: read 660 bytes LWP::Protocol::collect: read 32 bytes LWP::UserAgent::request: Simple response: OK [download] and the debugger stays alive and ready to proceed. On the off-chance this was an artifact of the debugger, I ran without the -d flag and sprinkled in some print statements. This didn't produce any change--LWP::Debug gave the same output and I died in the same place. So at this point, I'm slightly stumped. I suppose my next move is either to give up and say "What the heck do I care? It works, doesn't it?" or step laboriously through the LWP code itself to see where it dies. I'm certainly open to a better idea than either of those two. Afterthought: I'm running 5.6.1. LWP is version 5.63. adamsj They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen	[reply] [d/l] [select]
Re: LWP oddity by dws (Chancellor) on Nov 13, 2002 at 00:22 UTC
They've added us to the allow list and should be getting through, but we're not. Until you get clarity on how the "allow list" works, you're going to suffer on this one. Are they allowing by IP address (in which case little you do at your end is going to make a difference), or are they allowing by browser type (which you can forge)? And are you really sure you want to be proxying 'ftp' and not 'http'? I've had little luck using FTP through firewalls, unless passive FTP is supported by the remote server. YMMV.	[reply]
Re: Re: LWP oddity by adamsj (Hermit) on Nov 13, 2002 at 18:24 UTC
It does look odd, doesn't it? But I recall tinkering with the proxy statement for quite a while before getting a working combination with `$agent->proxy('ftp' => 'http://www.nunya.biz:8080');` [download] Now that the proxy server has been added to the allow list, I'm getting my files just fine. From my boss's point of view, the problem is fixed--but I still want to know why I'm silently dying when I remove the proxy statement! adamsj They laughed at Joan of Arc, but she went right ahead and built it. --Gracie Allen	[reply] [d/l]