in reply to WWW::Mechanize with https and a proxy

First you don't need use Crypt::SSLeay; -- LWP will do all job for you.

Second i think $mech->proxy('https', 'http', 'ftp', 'my_proxy:8080'); much better than $ENV{HTTPS_PROXY} = 'my_proxy:8080';

And third i think your code is good. So you right (IMHO) this is your proxy server issue and not your code.

  • Comment on Re: WWW::Mechanize with https and a proxy

Replies are listed 'Best First'.
Re^2: WWW::Mechanize with https and a proxy
by grinder (Bishop) on Oct 10, 2007 at 06:07 UTC
    First you don't need use Crypt::SSLeay; -- LWP will do all job for you.

    Nope, that's not true. LWP delegates the handling of setting up the SSL connection to Crypt::SSLeay, or, more specifically, the Net::SSL package therein (which LWP will load on demand). These days, Crypt::SSLeay itself is an empty shell.

    On the possibility that I could be misunderstanding you, let me put it another way: yes, the declaration is not needed in the client code, but the Crypt-SSLeay distribution must be installed for an https connection to work.

    While you may not like the environment variable hack, it's about the only way to transmit out-of-band information to a delegated-to package, without polluting the call stack all the way down. I suppose some sort of API could be added at the class level, but it would probably be tricky to get right when threads are taken into account.

    • another intruder with the mooring in the heart of the Perl

      Of course i mean declaration.

Re^2: WWW::Mechanize with https and a proxy
by almut (Canon) on Oct 10, 2007 at 06:21 UTC
    ... $mech->proxy(['https', 'http', 'ftp'], 'my_proxy:8080'); much better than $ENV{HTTPS_PROXY} = 'my_proxy:8080';

    You are correct in theory :)  In practice, though, I think it's a known problem that LWP's proxy method doesn't work with HTTPS (see the yellow box on the right on the page linked to). This has come up a couple of times (here at PM, and elsewhere), and the workaround has so far typically been to use Crypt::SSLeay in combination with the HTTPS_PROXY env var. So, unless the issue has been fixed in the meantime (which I don't think it has), it's always a good default strategy to try what has worked for others...  Hopefully, the OP will report back with what worked in this case.

      Just an FYI, I've found I definitly need the SSLeay and env variable setting, otherwise it hangs (so I can confirm the above statement). At least I get something back when I include them, even if it is a 400 error. If this script works for others without a proxy, then I can only assume it's something wonky with our proxy, which is a bummer. I'll ping the internal help group again and report back if there's an update.

        OK, I just spent two hours figuring this out. Make sure you are using ssleay (not openssl)

        Here is the code to properly setup mechanize to use a private proxy:

        use WWW::Mechanize;
        my $proxy='proxy.host.net:8080';
        my $proxy_user='proxy_username';
        my $proxy_pass='proxy_password';
        $ENV{HTTPS_PROXY} = 'http://'.$proxy;
        $ENV{HTTPS_PROXY_USERNAME} = $proxy_user;
        $ENV{HTTPS_PROXY_PASSWORD} = $proxy_pass;
        my $mech = WWW::Mechanize->new();
        $mech->agent('Mozilla/5.0');
        $mech->proxy('http', 'http://'.$proxy_user.':'.$proxy_pass.'@'.$proxy);
        $mech->proxy('https', undef);
        $mech->get('https://www.somehost.com/');
        

      As i remember i work fine with LWP through proxy (HTTP) with some Google pages (https://adwords.google.com/select/main?cmd=KeywordSandbox). As i remember LWP can't work with SOCKS proxy :-(. And for such task i use Crypt::SSLeay.