ultranerds has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

Got this annoying thing I've been battling with for the last hour or so :(

Basically - the user account can provide a URL in the format of http://user:pass@domain.com/file.csv, OR just http://domain.com/file.csv.

The code below checks to see if there is an @ in the URL, and if so it will try and use the authorization_basic() function of HTTP::Request.

It works perfectly when not trying to do user:pass , but when you include a password in the URL it doesn't work. Here is the code: 

    my $url_to_get = $_[1] || $details->{client_detail_url_list_offers
+};

     my $ua = LWP::UserAgent->new;
        $ua->agent("Import Script for VoyageForum");
        $ua->timeout(10);
        #$ua->env_proxy;
        
    # if we have an @ in the URL, they will be giving us something lik
+e ftp://someone:somewhere@doamin.com .. so we need to process this ap
+propriatly..
     my $request;
     if ($url_to_get =~ /\@/) {
        my @split = split /\@/, $url_to_get; # [ftp://user:pass] @ [do
+main.com]
        # ok, lets try and get the bit after //, so we have the user/p
+ass combo...
        if ($split[0] =~ m|(.*)//(.*)|) { 
            print $IN->header;
            #print qq|Got user/pass combo of: $2 \n|;
            my ($user,$pass)  = split /\:/, $2;
            
            $request = HTTP::Request->new(GET => qq|$1/$split[1]|);
            $request->authorization_basic($user, $pass);
            
            print qq|Bla, got URL of: "$1//$split[1]" <br />user: $use
+r <br />pass: $pass <br /><br />|;

    
            #exit;
        }
     } else {
        $request = HTTP::Request->new(GET => $url_to_get);
     }
        
     my $response = $ua->request($request);

[download]
..and I get this error: 
Bla, got URL of: "http://domain.org/illicotravel_import/marmara.txt"
user: dev
pass: password

[download]


(which is all exactly right)... yet I get this error:

Error: 500 No Host option provided

Anyone got any ideas? I'm coming up blank here :(

TIA!

Andy

Replies are listed 'Best First'.
Re: LWP::Useragent - why doesn't this code work? :/
by BrowserUk (Patriarch) on Jun 14, 2010 at 14:43 UTC

    At point ## 3 ##, you're relying on the value of $1 captured at point ## 1 ##; but you've re-used the regex engine in the split command at point ## 2 ##, which has probably overwritten it. Save the value of the captures immediately after they are captured. 

            if ($split[0] =~ m|(.*)//(.*)|) {        ## 1 ##
            print $IN->header;
            #print qq|Got user/pass combo of: $2 \n|;
            my ($user,$pass)  = split /\:/, $2;      ## 2 ##
            
            $request = HTTP::Request->new(GET => qq|$1/$split[1]|);   
+## 3 ##

    [download]
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
      Legend - that works! Thanks a ton!

      For anyone who may have the same issue, then please try: 

          # if we have an @ in the URL, they will be giving us something lik
+e ftp://someone:somewhere@doamin.com .. so we need to process this ap
+propriatly..
     my $request;
     if ($url_to_get =~ /\@/) {
        my @split = split /\@/, $url_to_get; # [ftp://user:pass] @ [do
+main.com]
        # ok, lets try and get the bit after //, so we have the user/p
+ass combo...
        if ($split[0] =~ m|(.*)//(.*)|) { 
            
            my $URI = qq|$1//$split[1]|;
            
            print $IN->header;
            #print qq|Got user/pass combo of: $2 \n|;
            my ($user,$pass)  = split /\:/, $2;
            
            
            $request = HTTP::Request->new(GET => $URI);
            $request->authorization_basic($user, $pass);
            
            print qq|Bla, got URL of: "$URI" <br />user: $user <br />p
+ass: $pass <br /><br />|;

    
            #exit;
        }
     } else {
        $request = HTTP::Request->new(GET => $url_to_get);
     }

      [download]
[reply]
[d/l]

        ...but if the value of $1 was wrong (overwritten) at ## 3 ##, why did it revert to the correct value (http:) in the subsequent debug print in your original code??   Sure, it's good practice to store away captures early, but not doing so is unlikely to have been the issue in this particular case.

[reply]
[d/l]
[select]
Re: LWP::Useragent - why doesn't this code work? :/
by almut (Canon) on Jun 14, 2010 at 14:43 UTC 
    $request = HTTP::Request->new(GET => qq|$1/$split[1]|)
                                          ^

    [download]

    The URL you GET (one slash http:/...) is not the one you print out two lines below (two slashes http://...).

    "No Host option provided" is the error you get when starting an URL with http:/domain.org/...

[reply]
[d/l]
[select]
      Yeah, already caught that one - but still not working :(
[reply]
Re: LWP::Useragent - why doesn't this code work? :/
by ikegami (Patriarch) on Jun 14, 2010 at 16:08 UTC
    It sounds like all of your problems would have been avoided if you had used URI to parse the URL.
[reply]

Back to Seekers of Perl Wisdom