iamsteve has asked for the wisdom of the Perl Monks concerning the following question:

Ok, 2 questions to annoy the advanced perl monk.

1)Anyone know why a "clicked" link from a site using "https" is not showing up in the $ENV{HTTP_REFERER} variable?
(eg. https://perl.com -users clicks on hyperlink to http://perlmonkks.org but sends no REFERER Variable)

2)Does anyone know a way of using LWP, HTTP::REQUEST, Basic credentials to obtain a web page if its using "https"? I want to check mail setting the URL and "credentials" and send it to my screen but, the URL is "https://"



This is sample code for question 2.

#!/usr/bin/perl use LWP; use HTTP::Headers; $myurl = "https://www.somewhereoutthereusinghttps.com/blah"; $h = new HTTP::Headers(Referer => $myurl); my $ua = LWP::UserAgent->new; my $req = HTTP::Request->new(GET, $myurl, $h); $req->authorization_basic('what', 'password'); my $response = $ua->request($req); print $response->headers_as_string; print "\n"; print $response->content;

It seems to either give an error or the screen is blank

Replies are listed 'Best First'.
Re: LWP,REQUEST,Credentials,Authentication,Grabbing Web Page
by joealba (Hermit) on Nov 13, 2001 at 17:39 UTC
    From Web Techniques: Referer Refresher

    According to Microsoft knowledge base article number Q178066, IE suppresses the Referer field whenever there's a transition between two protocol types, such as https: to http:, or file: to http:.

    1 down.. 1 to go. :)
Re: LWP,REQUEST,Credentials,Authentication,Grabbing Web Page
by cacharbe (Curate) on Nov 13, 2001 at 18:11 UTC
    SuperSearch is your friend. The answer is here, on this site. A couple times over, actually. This probably sums it up the best. Too, you'll want to look at using basic authentication. I've used it (right now as a matter of fact) it works.

    C-.