in reply to Re^2: How to Save Fetched Web Files as "path/$string.xml"
in thread How to Save Fetched Web Files as "path/$string.xml"

The following sample may help clear up the interpolation issue. Don't worry too much about the first few lines. The key stuff is inside the for loop. The regex pulls the two parts out of the URL that you want to munge together and the double quoted string does the munging by interpolating the contents of the variables into the string. Note the use of the {} to let the Perl interpreter know that the trailing _ is not part of the variable name.

use warnings;
use strict;

my $root = 'http://gd2.mlb.com/components/game/aaa/year_2007/month_08/
+day_06/gid_2007_08_06_quiaaa_yucaaa_1/batters/';
my @batters = map {"$root$_"} qw(112039.xml  120107.xml);

for my $url (@batters) {
    my ($prefix, $file) = $url =~ m!/(gid\w*)/batters/(.*)!;
    print "${prefix}_$file\n";
}

[download]

Prints:

gid_2007_08_06_quiaaa_yucaaa_1_112039.xml
gid_2007_08_06_quiaaa_yucaaa_1_120107.xml

[download]

The double quoted string following the print can drop into your open, but remove the \n at the end and remember to use the three parameter version and to check the result.

DWIM is Perl's answer to Gödel

Replies are listed 'Best First'.
Re^4: How to Save Fetched Web Files as "path/$string.xml"
by nase (Novice) on Aug 19, 2007 at 07:17 UTC

    This post now contains the actual code not previously provided at the bottom.

    I have not yet been able to work with the script for those files specifically (that one is next). However, I am trying to apply the interpolation method to a similar script (in scratchpad) that I've already built (using the same gid URL). Please note that I am unable to 'use strict' as it is presenting too many conflicts with the time::local module I'm using in another part of the code.

    This 3-parameter open works very well as you suggested but isn't the desired result: open(FILEHANDLE, ">","$outputdir/players.xml")

    This does not work: open(FILEHANDLE, ">","$outputdir/$game._players.xml")

    I'm trying to open as a filename such as $outputdir/$game_players.xml where $game="gid...." The only difference from the last problem is that '110246' is now 'players'. I've tried my best to adapt the code but I keep hitting roadblocks. The important part of the code is now in my scratchpad. Note that I am using LWP to pull data from the web.

    Thanks as always. Edit: The warning that I am getting is the 'could not open' one at the bottom of the code. I also read a little bit about the File::Spec module which has something to do with directories and filenames but I was unable to find anything on it past the brief descriptions in the Perl manual. Could be irrelevant.

    my $sourceurl = "http://gd2.mlb.com/components/game/aaa";
my $outputdir = "./aaa_players";
my $dayurl = "$sourceurl/year_$year/month_$mon/day_$mday/";

    my $response = $browser->get($dayurl); 
    die "Couldn't get $dayurl: ", $response->status_line, "\n"
        unless $response->is_success; 
    my $html = $response->content;
    my @games = @_;
        while($html =~ m/<a href=\"(gid_\w+\/)\"/g ){
        push @games, $1;}

# the loop that downloads data
    foreach $game (@games) {
    my $gameurl = "$dayurl/$game";
        $response = $browser->get($gameurl);
        die "Couldn't get $gameurl: ", $response->status_line, "\n"
        unless $response->is_success;
        $gamehtml = $response->content;
        if($gamehtml =~ m/<a href=\"players\.xml\"/ ) {
        $plyrurl = "$dayurl/$game/players.xml";
        $response = $browser->get($plyrurl);
        die "Couldn't get $plyrurl: ", $response->status_line, "\n"
            unless $response->is_success;
        $plyrhtml = $response->content;       
        print "\t\tfetching game: $game\n";
           open(FILEHANDLE, ">","$outputdir/players.xml") 
            or die "could not open file $game/players.xml: $|\n";    
           print FILEHANDLE "$game\n";
           close FILEHANDLE;
        } else {
        print "warning: no player list for $game\n";
        }

    [download]
[reply]
[d/l]
[select]

      Always use strict;. If that reports errors you don't understand, reduce the code to a simple example that shows the error and ask here about it. There are very few situations where you are ever likely to need to turn strictures off, and even for most of those there are likely to be better ways to achieve what you want.

      Avoid referring to code on your scratch pad - it's transient but the node that referees to it will be around for a while. Include the code with your node.

      In your scratch pad code you have:

      print "\t\tfetching game: $game\n";
open(FILEHANDLE, ">","$outputdir/players.xml")
    or die "could not open file $game/players.xml: $|\n";    
print FILEHANDLE "$game\n";

      [download]

      which confuses me because you say you are opening a different file than that which you pass into the open. I'd be inclined to do something like:

      my $filename = "$outputdir/players.xml";
print "\t\tfetching game: $game\n";
open(FILEHANDLE, ">", $filename)
    or die "could not open file $filename: $!\n";    
print FILEHANDLE "$game\n";

      [download]

      BTW, $! is the special variable that contains the last OS error.

      DWIM is Perl's answer to Gödel
[reply]
[d/l]
[select]

        Thanks again for your patience with this. I believe that I have successfully made the code strict with the help of others here. I just want to re-iterate what I am trying to achieve:

        In reference to the LWP module-based code excerpt below, I am trying to pull the files represented by my $plyrurl (line 24) below and save them as "$outputdir/$game_$players" represented by my $filename(line 31) (so a file might save as "$outputdir/gid_2007_08_06_quiaaa_yucaaa_1_112039.xml"). One point of confusion for me is that the original code author pulls content from 2 pages on his way to the destination page (line 9 and 22). This type of code is called a "spider" so maybe the code needs to pull data from each page on its path to the destination page? I just think it would be easier to find a solution with simpler code if anything can be taken out!

        That said, I tried to adapt your original reply using the batters example to this code but am getting lost with the character class abbreviations. Because of that, I am trying to concatenate the strings instead. The current code here is returning a "could not open file ./xxx_players/gid_2007_08_06_quiaaa_yucaaa_1/_112039.xml No such file or directory" error. Shouldn't the last part not have the backslash there before '_112039'?

        my $sourceurl = "http://gd2.mlb.com/components/game/aaa";
my $outputdir = "./xxx_players";
my $dayurl = "$sourceurl/year_$year/month_$mon/day_$mday/";
    print "\t$dayurl\n";
    
my $response = $browser->get($dayurl); 
    die "Couldn't get $dayurl: ", $response->status_line,   
        "\n" unless $response->is_success;
my $html = $response->content;
my @games = @_;
    while($html =~ m/<a href=\"(gid_\w+\/)\"/g ){  
        push @games, $1;}

# the loop that downloads data
my $game; 
foreach $game (@games) {
    my $gameurl = "$dayurl/$game";
    $response = $browser->get($gameurl);
        die "Couldn't get $gameurl: ",                
            $response->status_line, "\n" unless         
                $response->is_success;
    my $gamehtml = $response->content;
    if($gamehtml =~ m/<a href=\"players\.xml\"/ ) {
        my $plyrurl = "$dayurl/$game/players.xml";
            $response = $browser->get($plyrurl);
        die "Couldn't get $plyrurl: ",  
                    $response->status_line, "\n"
                unless $response->is_success;
        my $plyrhtml = $response->content;
        my $players = 'players.xml';
        my $filename = "$outputdir/$game" . "$players";
        print "\t\tfetching game: ${game}_$players\n";
        open(FILEHANDLE, ">","$filename"
                or die "could not open file $filename $!\n";    
        print FILEHANDLE "$game" . "$filename";
        close FILEHANDLE;
    } else 
       {my $players = 'players.xml';
       print "warning: no player list for $game . $players\n";
    }

        [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom