syphilis has asked for the wisdom of the Perl Monks concerning the following question:

Hi,
On Ubuntu: 
$ wget http://www.cpan.org/authors/01mailrc.txt.gz
URL transformed to HTTPS due to an HSTS policy
--2019-11-21 21:21:48--  https://www.cpan.org/authors/01mailrc.txt.gz

[download]
Who is imposing that "HSTS policy" ? (It seems to have sprung up in the last day or two.)
That file, which is the first thing that my cpan utility tries to download whenever I try to install a module, can no longer be downloaded (by me, at least) over HTTP protocol ... and on Windows there's nothing automatically available to the cpan utility that can perform HTTPS downloads.

I've installed wget into MSYS2 on the Windows machine (as it's capable of https downloads).
How do I configure the cpan utility to use wget ?
And how do I specify that https://www.cpan.org/authors/01mailrc.txt.gz should be downloaded, not http://www.cpan.org/authors/01mailrc.txt.gz .

Cheers,
Rob

Replies are listed 'Best First'.
Re: HSTS policy breaks cpan utility on Windows
by marto (Cardinal) on Nov 21, 2019 at 13:01 UTC

    To set https as the default:

    cpan> o conf urllist
    urllist           
    0 [https://www.cpan.org]
cpan> o conf urllist pop
cpan> o conf urllist https://www.cpan.org
cpan> o conf commit

    [download]

    A patch for FirstTime.pm:

    --- FirstTime.pm.orig    2019-06-30 07:25:30.000000000 +0100
+++ FirstTime.pm    2019-11-21 12:45:34.742632425 +0000
@@ -1344,7 +1344,7 @@
             );
         }
         else {
-            $CPAN::Config->{urllist} = [ 'http://www.cpan.org/' ];
+            $CPAN::Config->{urllist} = [ 'https://www.cpan.org/' ];
         }
     }
     elsif (!$matcher || "urllist" =~ $matcher) {

    [download]

    Of course other parts of the module should be altered to reflect any changes to dependencies etc. I don't know what caused this to happen in the last few days for you. Perhaps some Windows update, GPO update or some network shenanigans?

    Update: an existing PR to address this in a more complete fashion.

[reply]
[d/l]
[select]
      cpan> o conf urllist https://www.cpan.org

      I had taken a look through MyConfig.pm, but failed to notice "urllist" and its setting.

      Amusingly, that alone didn't do the trick for me on Windows because HTTP::Tiny is used as the download agent - and it's not capable of https transfers on a fresh build of perl from source.
      Changing the "wget" setting to "wget" fixes that little problem.

      Thanks for the assistance !!

      Cheers,
      Rob
[reply]

        "Amusingly, that alone didn't do the trick for me on Windows because HTTP::Tiny is used as the download agent - and it's not capable of https transfers on a fresh build of perl from source."

        Ah, sorry, I didn't appreciate that this was a perl you'd built yourself. Strawberry (I've no idea about AS these days) ships with the required IO::Socket::SSL and Net::SSLeay for direct SSL/https support within HTTP::Tiny.

[reply]
Re: HSTS policy breaks cpan utility on Windows
by cavac (Prior) on Nov 21, 2019 at 14:15 UTC

    HSTS stands for "HTTP Strict Transport Security". Its defined in RFC6797

    It has about 50 pages, but the gist of it is "Webmasters can declare through HTTP headers that they want to protect their users and only allow secured connections". Which frankly is a very good idea - i'd probably throw a party if the IETF declares unencrypted traffic completely illegal and bans it from existance.

    perl -e 'use Crypt::Digest::SHA256 qw[sha256_hex]; print substr(sha256_hex("the Answer To Life, The Universe And Everything"), 6, 2), "\n";'
[reply]
[d/l]

      acronym explained ++

      Surely encryption is out of scope for HTTP. That's the reason for https. The user should declare which protocol they want to use, and the provider which are available, where required.

[reply]

        rt://130819. The end user may not be aware of the underlying issues, a https default makes sense, forcing those who understand the issue to make the change where appropriate.

[reply]
Re: HSTS policy breaks cpan utility on Windows
by Anonymous Monk on Nov 21, 2019 at 23:46 UTC

    Its the webserver 

    $ wget cpan.org
--2019-11-21 15:50:25--  http://cpan.org/
Resolving cpan.org... 147.75.38.240
Connecting to cpan.org|147.75.38.240|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://cpan.org/ [following]
--2019-11-21 15:50:25--  https://cpan.org/
Connecting to cpan.org|147.75.38.240|:443... connected.
....

    [download]
[reply]
[d/l]
Re: HSTS policy breaks cpan utility on Windows
by Anonymous Monk on Nov 21, 2019 at 16:24 UTC
    > Who is imposing

    The Safety Police won't be happy until programmers are mandated by law to wear a helmet while coding. What will they break next? Maybe no one ever noticed, until some new "geniuses" were born, that shebangs are a critical vulnerablity requiring immediate removal from all scripts...

[reply]
Re: HSTS policy breaks cpan utility on Windows
by Anonymous Monk on Nov 22, 2019 at 20:00 UTC
    Still, obviously, the cpan utility must be promptly adjusted to handle this "graciously."
[reply]

Back to Seekers of Perl Wisdom