Your browser has a proxy set, or better network connectivity, than Perl.

Inspect your browser network settings closely (likely, the "Network" tab), and configure LWP appropriately. Most likely by setting $ENV{HTTP_PROXY} and $ENV{HTTPS_PROXY}, preferrably even before your Perl process starts. Especially for HTTPS, OpenSSL / SSLeay like to have first grab at these variables.

As you can see, the LWP failing url works fine in my browser and it has no proxies configured.

Nor do I have any proxies set in my environment:

C:\test>set proc
PROCESSOR_ARCHITECTURE=x86
PROCESSOR_ARCHITEW6432=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 15 Stepping 11, GenuineInt
+el
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=0f0b

C:\test>set http
Environment variable http not defined
[download]

The weird thing is that the same urls that fail (instantly and multiple times) will subsequently succeed occasionally. But in my browser, they always succeed.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

I would use WireShark to find out what is happening on the network layer. As your browser knows how to connect, but Perl doesn't, something in their behaviour is different. Maybe using WireShark, you can find out how their behaviour differs.

I think you've mentioned using Opera, and I also remember that Opera (Mini) has a Turbo Mode, where it uses an Opera-operated proxy. This might not show up in your general proxy settings.

Also consider whether Opera gets a different IP address for search.cpan.org than what Perl gets - Perl might end up with an unavailable mirror (sometimes) while Opera might be lucky due to longer DNS caching within the browser (instead of relying on the OS to provide "current" DNS).

Fetching with LWP:
http://www.cpan.org/authors/01mailrc.txt.gz
[download]

Taking out cpan.strawberryperl.com from the urllist avoids the redirect, but still the LWP get from cpan.pair.com fails with status500; but the fallback to wget succeeds:

C:\test>cpan

cpan shell -- CPAN exploration and modules installation (v2.05)
Enter 'h' for help.

cpan> o conf urllist
    urllist
        0 [http://cpan.strawberryperl.com/]
        1 [http://www.cpan.org/]
Type 'o conf' to view all configuration items


cpan> o conf urllist shift
commit: wrote 'C:\perl5.18\perl\lib/CPAN/Config.pm'

cpan> o conf urllist
    urllist
        0 [http://www.cpan.org/]
Type 'o conf' to view all configuration items


cpan> reload index
Fetching with LWP:
http://www.cpan.org/authors/01mailrc.txt.gz
LWP failed with code[500] message[Can't connect to www.cpan.org:80]

Trying with
    c:/perl64/bin/wget.exe -O "C:\Perl5.18\cpan\sources\authors\01mail
+rc.txt.gz.tmp7328"
to get
    http://www.cpan.org/authors/01mailrc.txt.gz
--2014-12-31 00:22:29--  http://www.cpan.org/authors/01mailrc.txt.gz
Resolving www.cpan.org... 94.242.223.198
Connecting to www.cpan.org|94.242.223.198|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 228720 (223K) [application/x-gzip]
Saving to: `C:/Perl5.18/cpan/sources/authors/01mailrc.txt.gz.tmp7328'

100%[===============================================================>]
+ 228,720     --.-K/s   in 0.1s

2014-12-31 00:22:30 (1.56 MB/s) - `C:/Perl5.18/cpan/sources/authors/01
+mailrc.txt.gz.tmp7328' saved [228720/228720]

Fetching with LWP:
http://www.cpan.org/modules/02packages.details.txt.gz
LWP failed with code[500] message[Can't connect to www.cpan.org:80]

Trying with
    c:/perl64/bin/wget.exe -O "C:\Perl5.18\cpan\sources\modules\02pack
+ages.details.txt.gz.tmp7328"
to get
    http://www.cpan.org/modules/02packages.details.txt.gz
--2014-12-31 00:22:30--  http://www.cpan.org/modules/02packages.detail
+s.txt.gz
Resolving www.cpan.org... 94.242.223.198
Connecting to www.cpan.org|94.242.223.198|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1615484 (1.5M) [application/x-gzip]
Saving to: `C:/Perl5.18/cpan/sources/modules/02packages.details.txt.gz
+.tmp7328'

100%[===============================================================>]
+ 1,615,484   2.11M/s   in 0.7s

2014-12-31 00:22:31 (2.11 MB/s) - `C:/Perl5.18/cpan/sources/modules/02
+packages.details.txt.gz.tmp7328' saved [1615484/1615484]

Fetching with LWP:
http://www.cpan.org/modules/03modlist.data.gz
LWP failed with code[500] message[Can't connect to www.cpan.org:80]

Trying with
    c:/perl64/bin/wget.exe -O "C:\Perl5.18\cpan\sources\modules\03modl
+ist.data.gz.tmp7328"
to get
    http://www.cpan.org/modules/03modlist.data.gz
--2014-12-31 00:22:31--  http://www.cpan.org/modules/03modlist.data.gz
Resolving www.cpan.org... 94.242.223.198
Connecting to www.cpan.org|94.242.223.198|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 248 [application/x-gzip]
Saving to: `C:/Perl5.18/cpan/sources/modules/03modlist.data.gz.tmp7328
+'

100%[===============================================================>]
+ 248         --.-K/s   in 0s

2014-12-31 00:22:31 (2.55 MB/s) - `C:/Perl5.18/cpan/sources/modules/03
+modlist.data.gz.tmp7328' saved [248/248]

Database was generated on Wed, 31 Dec 2014 00:20:56 GMT

Updating database file ...
[download]

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Basically, why would LWP fail repeatedly from many sources most of the time but occassionally succeed?

What do you get if you try perl -S -MDevel::VersionDump lwp-request -UuSsEd url on some of those urls?

Notice that with LWP the redirect to pair.com fails with 500; but succeeds with wget. It looks for all the world like pair.com is rejecting LWP's useragent string?

C:\test>perl -S -MDevel::VersionDump lwp-request -Ed http://cpan.straw
+berryperl.com/authors/01mailrc.txt.gz
Perl version: v5.18.4 on MSWin32 (C:\Perl5.18\perl\bin\perl.exe)
Carp               -   1.3301
Config             - 5.018004
Config_git.pl      -  Unknown
Config_heavy.pl    -  Unknown
Devel::VersionDump -     0.02
DynaLoader         -     1.18
Encode             -     2.62
Encode::Alias      -     2.18
Encode::Byte       -     2.04
Encode::Config     -     2.05
Encode::Encoding   -     2.07
Encode::Locale     -     1.03
Exporter           -     5.70
Exporter::Heavy    -     5.70
File::Basename     -     2.84
Getopt::Long       -     2.42
HTTP::Date         -     6.02
HTTP::Status       -     6.03
List::Util         -     1.41
Scalar::Util       -     1.41
Time::Local        -   1.2300
URI                -     1.64
URI::Escape        -     3.31
URI::Heuristic     -     4.20
Win32::API         -     0.79
Win32::API::Struct -     0.65
Win32::API::Type   -     0.69
XSLoader           -     0.16
base               -     2.18
bytes              -     1.04
constant           -     1.27
overload           -     1.22
overloading        -     0.02
strict             -     1.07
vars               -     1.03
warnings           -     1.18
warnings::register -     1.02
GET http://cpan.strawberryperl.com/authors/01mailrc.txt.gz
User-Agent: lwp-request/6.03 libwww-perl/6.08

302 Found
Connection: close
Date: Wed, 31 Dec 2014 10:30:00 GMT
Location: http://cpan.pair.com/authors/01mailrc.txt.gz
Server: Apache
Vary: Accept-Encoding
Content-Length: 228
Content-Type: text/html; charset=iso-8859-1
Client-Date: Wed, 31 Dec 2014 10:30:22 GMT
Client-Peer: 69.163.152.144:80
Client-Response-Num: 1
Title: 302 Found

GET http://cpan.pair.com/authors/01mailrc.txt.gz
User-Agent: lwp-request/6.03 libwww-perl/6.08

500 Can't connect to cpan.pair.com:80
Content-Type: text/plain
Client-Date: Wed, 31 Dec 2014 10:30:22 GMT
Client-Warning: Internal response

C:\test>wget http://cpan.pair.com/authors/01mailrc.txt.gz
--2014-12-31 10:31:47--  http://cpan.pair.com/authors/01mailrc.txt.gz
Resolving cpan.pair.com... 66.39.76.93
Connecting to cpan.pair.com|66.39.76.93|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 228720 (223K) [application/x-gzip]
Saving to: `01mailrc.txt.gz'

100%[=====================>] 228,720      240K/s   in 0.9s

2014-12-31 10:31:49 (240 KB/s) - `01mailrc.txt.gz' saved [228720/22872
+0]


C:\test>
[download]

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

d'oh, that doesnt give LWP version info ... seems to work here just fine

$ perl -S -MDevel::ModuleDumper lwp-download http://cpan.strawberryper
+l.com/authors/01mailrc.txt.gz
Saving to '01mailrc.txt.gz'...
223 KB received in 0 seconds (288 KB/sec)
Perl    5.016001
Compress::Raw::Zlib     2.063
Encode  2.55
Encode::Alias   2.18
Encode::Byte    2.04
Encode::Config  2.05
Encode::Encoding        2.07
Encode::Locale  1.03
Errno   1.15
Fcntl   1.11
File::Basename  2.84
File::Glob      1.17
File::GlobMapper        1.000
Getopt::Std     1.07
HTML::Entities  3.69
HTML::HeadParser        3.69
HTML::Parser    3.7
HTTP::Config    6.00
HTTP::Date      6.02
HTTP::Headers   6.05
HTTP::Message   6.06
HTTP::Request   6.00
HTTP::Response  6.04
HTTP::Status    6.03
IO      1.25_06
IO::Compress::Base::Common      2.063
IO::Compress::Gzip::Constants   2.063
IO::Compress::Zlib::Extra       2.063
IO::File        1.16
IO::Handle      1.33
IO::Seekable    1.1
IO::Socket      1.34
IO::Socket::INET        1.33
IO::Socket::IP  0.31
IO::Socket::UNIX        1.24
IO::Uncompress::Adapter::Inflate        2.063
IO::Uncompress::Base    2.063
IO::Uncompress::Gunzip  2.063
IO::Uncompress::RawInflate      2.063
List::Util      1.33
LWP     6.08
LWP::ConnCache  6.02
LWP::MediaTypes 6.02
LWP::MemberMixin        none
LWP::Protocol   6.06
LWP::Protocol::http     none
LWP::UserAgent  6.06
Net::HTTP       6.07
Net::HTTP::Methods      6.07
POSIX   1.30
Scalar::Util    1.33
SelectSaver     1.02
Socket  2.006
Storable        2.39
Symbol  1.07
Tie::Hash       1.04
Time::Local     1.2300
URI     1.60
URI::Escape     3.31
URI::http       none
URI::_generic   none
URI::_idna      none
URI::_punycode  0.04
URI::_query     none
URI::_server    none
Win32::API      0.75
Win32::API::Struct      0.64
Win32::API::Type        0.67
Win32::Console  0.09
[download]