HTTP::Request -Variable values not translating in URL

Tonyd5915 has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to download the contents of a web page. I have two parameter variables that I am trying to use in the URL:

$month = "1"
$year = "2013"
[download]

The problem is that the cookie values for the URL aren't being set properly. I then use the cookie data for a follow-up HTTP request. The URL works if I don't use variables and hard-code the values for Month and Year, but it doesn't work when I use the variables Month=$month and Year=$year. The variable is $url_to_get. It works when I put the values in for Month and Year:


$url_to_get = 'http://tennislink.usta.com/Tournaments/Schedule/SearchR
+esults.aspx?Action=2&Month=1&Year=2013&Sanctioned=0&SearchRadius=-1&E
+ntryLevel=False&Intermediate=False&Advanced=False';
[download]

but this doesn't work:


$url_to_get = 'http://tennislink.usta.com/Tournaments/Schedule/SearchR
+esults.aspx?Action=2&Month=$month&Year=$year&Sanctioned=0&SearchRadiu
+s=-1&EntryLevel=False&Intermediate=False&Advanced=False';
[download]

This is a portion of my script: (some of the modules are for other parts of the script)


#!/usr/bin/perl

use URI::URL;
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Headers;
use HTTP::Response;
use HTTP::Cookies;
use WWW::Mechanize;

$month = "3";
$year = "2014";

$url_to_get = 'http://tennislink.usta.com/Tournaments/Schedule/SearchR
+esults.aspx?Action=2&Month=$month&Year=$year&Sanctioned=0&SearchRadiu
+s=-1&EntryLevel=False&Intermediate=False&Advanced=False';

$ua = LWP::UserAgent->new;
$ua->agent('Mozilla/4.0 (Compatable; MSIE 5.01; Windows NT 5.0)');
$ua->cookie_jar(HTTP::Cookies->new(file => "lwpcookies.txt", autosave 
+=> 1));
$res = $ua->request(HTTP::Request->new(GET => "$url_to_get"));
$content = $res->as_string( );
@content = split ("\n", $content);

        foreach $info (@content)
        
        {

        if ($info =~ "Displaying 1 ")
            
            {
                print "$info\n";
            }

        }
[download]

This is what gets written to the cookie file:

cat lwp*
#LWP-Cookies-1.0
Set-Cookie3: BigIPCookie=3658774282.20480.0000; path="/"; domain=tenni
+slink.usta.com; path_spec; expires="2013-12-12 23:07:09Z"; version=0
Set-Cookie3: tennislink.usta.com_Search="Action=2&Keyword=&TournamentI
+D=&SectionDistrict=&City=&State=&Zip=&Month=0&Day=&Year=0&Division=&C
+ategory=&Surface=&OnlineEntry=0&DrawsSheets=0&PageNum=0&UserTime=&San
+ctioned=0&AgeGroup=&SkillLevel_EntryLevel=False&SkillLevel_Intermedia
+te=False&SkillLevel_Advanced=False&SearchRadius=-1&QuickSearch=0&Orde
+rBy=3"; path="/"; domain=tennislink.usta.com; path_spec; expires="201
+4-06-12 18:07:08Z"; version=0
[download]

You can see that the Month and Year values in the tennislink.usta.com cookie are set to zero. (The values for $month and $year are normally entered on the command line).

Any assistance will be greatly appreciated.

Comment on HTTP::Request -Variable values not translating in URL Select or Download Code

Replies are listed 'Best First'.
Re: HTTP::Request -Variable values not translating in URL by toolic (Bishop) on Dec 12, 2013 at 19:03 UTC
Single quotes prevent variable interpolation. Use double quotes: `$url_to_get = "http://tennislink.usta.com/Tournaments/Schedule/SearchR +esults.aspx?Action=2&Month=$month&Year=$year&Sanctioned=0&SearchRadiu +s=-1&EntryLevel=False&Intermediate=False&Advanced=False";` [download]	[reply] [d/l]
Re^2: HTTP::Request -Variable values not translating in URL by Tonyd5915 (Initiate) on Dec 12, 2013 at 19:22 UTC
I have tried double quotes - it didn't help the cookie problem.	[reply]
Re^3: HTTP::Request -Variable values not translating in URL by Tonyd5915 (Initiate) on Dec 12, 2013 at 19:33 UTC
After the initial URL, I am trying to get the second URL which uses the information in the cookie to process the request: This is the second URL: `$url_to_get2 = "http://tennislink.usta.com/Tournaments/Schedule/prints +earchresults.aspx?print_all_one=1";` [download]	[reply] [d/l]
Re: HTTP::Request -Variable values not translating in URL by PerlSufi (Friar) on Dec 12, 2013 at 21:08 UTC
A while back, I had to write a script that crawled a site which distributed a file through a flash application. I noticed that the cookie was always contained in the url and followed a specific pattern. Thanks to PerlMonks, I was able to figure that out. Firstly, I used WWW::Mechanize::Firefox.. you may need to in this case, too- I'm not sure. Anyways I start with something like: `use strict; use warnings; use WWW::Mechanize::Firefox; use Data::Dumper; use DateTime; ## you can set the date to nearly anything with this module, refer to +its documentation my $mech = WWW::Mechanize::Firefox->new(launch => 'C:\path\to\your\firefox.exe', autoclose => 1, autodie => 1 ); my $jar = $mech->cookies(); print Dumper($jar); # for debugging, you need to understand the data s +tructure of this cookie to properly use it` [download] Alternatively, you may not need to use the cookies at all. But since it is not completely clear to me what your goal is with this script, I can't say for certain. Briefly looking at your url, you may be able to use just plain old WWW::Mechanize and then $mech->dump_text;	[reply] [d/l]
Re^2: HTTP::Request -Variable values not translating in URL by Tonyd5915 (Initiate) on Dec 12, 2013 at 22:28 UTC
I am an idiot. I had another variable named "$year" a couple hundred lines up the script... It was from a routine that captured the date. Thank goodness I only wasted six hours on this...	[reply]
Re^3: HTTP::Request -Variable values not translating in URL by Anonymous Monk on Dec 13, 2013 at 02:55 UTC
Read this if you want to cut your development time in half! goes hand-in-hand with Basic debugging checklist , brian's Guide to Solving Any Perl Problem, CGI Help Guide , Troubleshooting Perl CGI scripts ... More generic advice :) On debugging, verify everything, talk to teddybear ... checklists and more , perltrap, perltrap, Common Perl Pitfalls / perltrap)<c>	[reply]
Re: HTTP::Request -Variable values not translating in URL by taint (Chaplain) on Dec 12, 2013 at 21:00 UTC
I'm wondering if `$month` and `$year` are `undef` Just for fun try this `# add just below your shebang # always use strict use strict; # change the 2 entries you have to these my $month = ('3'); my $year = ('2014');` [download] all other things being the same. --Chris Yes. What say about me, is true.	[reply] [d/l]