in reply to HTTPS returning 400 from Linux but 200 OK from Windows

Consider looking at the actual data that gets exchanged via LWP::ConsoleLogger maybe.

I assume that your proxy wants a username+password, which you don't give it. Maybe curl is better or uses some different credentials there.

  • Comment on Re: HTTPS returning 400 from Linux but 200 OK from Windows

Replies are listed 'Best First'.
Re^2: HTTPS returning 400 from Linux but 200 OK from Windows
by Random_Walk (Prior) on Feb 13, 2017 at 18:54 UTC
[reply]
[d/l]
[reply]
Re^2: HTTPS returning 400 from Linux but 200 OK from Windows
by Random_Walk (Prior) on Jan 28, 2017 at 07:00 UTC

    I found a 5.18 perl on a network adjacent SuSE 12sp1 server

    me@linux:/home/rpollard> export HTTPS_PROXY='http://user:pass@proxy:80
+80'
me@linux:/home/rpollard> echo $HTTPS_PROXY
http://user:pass@proxy:8080

me@linux:/home/rpollard> curl https://www.google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset
+=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.se/?gfe_rd=cr&amp;ei=6T-MWJGBLsiq8weZ_YCIA
+Q">here</A>.
</BODY></HTML>

me@linux:/home/rpollard> perl GetTest.pl
400 Bad Request at GetTest.pl line 16.

me@linux:/home/rpollard> unset HTTPS_PROXY
me@linux:/home/rpollard> echo $HTTPS_PROXY

me@linux:/home/rpollard> curl https://www.google.com
curl: (6) Could not resolve host: www.google.com
me@linux:/home/rpollard> perl GetTest.pl
500 Can't connect to google.com:443 (Bad hostname) at GetTest.pl line 
+16.
me@linux:/home/rpollard> perl -v

This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-li
+nux-thread-multi

Copyright 1987-2013, Larry Wall

Perl may be copied only under the terms of either the Artistic License
+ or the
GNU General Public License, which may be found in the Perl 5 source ki
+t.

Complete documentation for Perl, including FAQ lists, should be found 
+on
this system using "man perl" or "perldoc perl".  If you have access to
+ the
Internet, point your browser at http://www.perl.org/, the Perl Home Pa
+ge.

    [download]
    That is using the same test script as in the root node. I think this shows both curl and perl are using the same HTTPS_PROXY here. Now I have a more modern Perl, I will install LWP::ConsoleLogger and see what that tells me.

    Cheers,
    R.

    Pereant, qui ante nos nostra dixerunt!
[reply]
[d/l]

      Server is all locked down, no gcc/cc etc and seos won't even let me install gcc, so I can't cpan LWP::ConsoleLogger. Now I can either go through the full change cycle to get a test server, or use curl in my scripts to pull back the data form the REST API I am after, then parse it from there. Neither choice appeals

      Cheers,
      R.

      Pereant, qui ante nos nostra dixerunt!
[reply]
[d/l]
Re^2: HTTPS returning 400 from Linux but 200 OK from Windows
by Random_Walk (Prior) on Jan 28, 2017 at 06:41 UTC

    Hi Corion,

    I have HTTP_PROXY and HTTPS_PROXY set with http://user:pass@proxyhost and these are used by curl so must be correct.

    I tried to install lwp::ConsoleLogger with CPAN, and it failed as my Perl is too old (5.10). I suspect this may be the cause, so I am having a quick look around for a server in the same data center with a newer Perl to test if that is the critical difference. The Perl on my lappy that works is 5.22.1.

    Cheers,
    R.

    Pereant, qui ante nos nostra dixerunt!
[reply]
Re^2: HTTPS returning 400 from Linux but 200 OK from Windows
by Random_Walk (Prior) on Feb 13, 2017 at 18:07 UTC

    I have LWP::ConsoleLogger installed after much struggle to get it built within the restrictions of the client site. Sadly it has brought me no new enlightenment. My only though now is that Perl does not like my https_proxy being an http url. 

    # curl https://google.com/
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset
+=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.se/?gfe_rd=cr&amp;ei=IvWhWPTvI4b67gSTn7ygA
+g">here</A>.
</BODY></HTML>

# perl test.pl
GET https://google.com/

.-----------------+--------------------.
| Request Header  | Value              |
+-----------------+--------------------+
| Accept-Encoding | gzip               |
| User-Agent      | WWW-Mechanize/1.83 |
'-----------------+--------------------'

==> 500 Can't connect to google.com:443

.-----------------+-------------------------------.
| Response Header | Value                         |
+-----------------+-------------------------------+
| Client-Date     | Mon, 13 Feb 2017 18:04:23 GMT |
| Client-Warning  | Internal response             |
| Content-Type    | text/plain                    |
'-----------------+-------------------------------'

.---------------------------------------------------------------------
+---------------------------------------------------------------------
+---.
| Content                                                             
+                                                                     
+   |
+---------------------------------------------------------------------
+---------------------------------------------------------------------
+---+
| Can't connect to google.com:443                                     
+                                                                     
+   |
|                                                                     
+                                                                     
+   |
| Name or service not known at /usr/lib/perl5/site_perl/5.18.2/LWP/Pro
+tocol/http.pm line 47, <DATA> line 1.                                
+   |
'---------------------------------------------------------------------
+---------------------------------------------------------------------
+---'

.---------------------------------------------------------------------
+---------------------------------------------------------------------
+---.
| Text                                                                
+                                                                     
+   |
+---------------------------------------------------------------------
+---------------------------------------------------------------------
+---+
| Can't connect to google.com:443                                     
+                                                                     
+   |
|                                                                     
+                                                                     
+   |
| Name or service not known at /usr/lib/perl5/site_perl/5.18.2/LWP/Pro
+tocol/http.pm line 47, <DATA> line 1.                                
+   |
'---------------------------------------------------------------------
+---------------------------------------------------------------------
+---'

Error GETing https://google.com/: Can't connect to google.com:443 at t
+est.pl line 22.

    [download]
    Thanks for taking the time. I have at least learned a bit more about the web tools available in Perl :)

    Cheers,
    R.

    Pereant, qui ante nos nostra dixerunt!
[reply]
[d/l]

      Unfortunately, LWP::ConsoleLogger doesn't seem to be talkative about when it is trying to connect to a proxy.

      You can try to find which proxy LWP will try to use:

      #!perl -w
use strict;
use LWP::ConsoleLogger::Easy 'debug_ua';
use Data::Dumper;
use LWP::UserAgent;

my $ua=LWP::UserAgent->new();
warn Dumper $ua->proxy('https');
my $logger=debug_ua($ua,5);
print Dumper \%ENV;
$ua->env_proxy;
warn Dumper $ua->proxy('https');
$ua->get('https://google.com')

      [download]

      This should show you that your proxy URL is in %ENV and also picked up from there in the $ua.

      Maybe your proxy does not let Perl through? Have you tried masquerading your Perl script as curl (because that seems to work?)?:

      $ua->default_headers->header('User-Agent' => 'curl/7.10.x (i386-redhat
+-linux-gnu) libcurl/7.10.x OpenSSL/0.9.7a ipv6 zlib/1.1.4');

      [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom