Flame has asked for the wisdom of the Perl Monks concerning the following question:

Using Windows ActiveState Perl 5.8.8 build 822.

I'm extremely confused.

I just tossed together a simple script that would scan the Wizards Gatherer web page and spit out a card's ID and name, separated by a semicolon.

Now I'm a little out of practice, but I managed to get it to work first try, printing to STDOUT. I then said to myself "okay, why bother with the file operations, let the shell do it."

perl GatherScan.pl > carddb.txt

I come back in a few minutes to find no data in carddb.txt. It's reporting a filesize of 0.

I reworked it using IO::File to do the file writes itself. Thus adding 

use IO::File;
my $file = IO::File->new('>carddb.txt') or die $!;

[download]
to the front of my file and changing my print invocations to $file->print.

That done, I ran it again, without the redirect, and found, again, no data in the file. I tried changing the file name to no avail. I added prints to STDERR to verify the data was being retrieved/generated, and I got exactly what I wanted out of it, but no actual data in the file.

Finally, I tried redirecting STDERR to the file and swapping everything (by this time I'd given up on having perl write to the file directly.) Suddenly, the file has data!

Here's the program I ended up with, with the swap completed: 

use LWP::Simple 'get';
use strict;
use warnings;

my $basePath = "http://ww2.wizards.com/gatherer/CardDetails.aspx?id=";

for(my $i = 1; $i < 129700; ++$i){
    print STDOUT "Getting $i ... ";
    my $content = get($basePath.$i);
    print STDOUT "Done\n";
    if($content){
        $content =~ m/<span id="_lblCardTitle">((?:\w|\s)+)<\/span>/;
        my $result = $i.';'.$1."\n"; 
        print STDERR $result;
        print STDOUT $result;
    }else{
        print STDOUT 'Failed on ', $i, "\n";
    }
    
    sleep 1;
}

print STDOUT "Done!\n";

[download]

As you can see, it was meant to be a simple little script, doesn't even do proper HTML parsing (I know).

So, the question is, does anyone know why I couldn't get data from STDOUT to a file.. or into a file opened in perl?

Thanks.

0x596F752068617665206265656E2068657865642E

Replies are listed 'Best First'.
Re: Can't write files (even redirected from shell)
by ikegami (Patriarch) on Jan 14, 2008 at 04:13 UTC
    When you checked the size of the file, was the program still running? File handles other than STDERR are buffered by default. Add 
    use IO::Handle qw( );
STDOUT->autoflush(1);

    [download]
[reply]
[d/l]

      ... Thank you... I knew it would be something simple...

      Let this be a lesson to ALL!! Don't fall out of perl practice... I used to know that...

      Again, thanks.

      0x596F752068617665206265656E2068657865642E

[reply]

        Is Morningtide out yet? Guess not.

        For long running processes like that, I usually tee the output. And since Windows doesn't have tee, I wrote one quickly.

        tee.bat: 

        @rem = '--*-Perl-*--
@echo off
if "%OS%" == "Windows_NT" goto WinNT
perl -x -S "%0" %1 %2 %3 %4 %5 %6 %7 %8 %9
goto endofperl
:WinNT
perl -x -S %0 %*
if NOT "%COMSPEC%" == "%SystemRoot%\system32\cmd.exe" goto endofperl
if %errorlevel% == 9009 echo You do not have Perl in your PATH.
if errorlevel 1 goto script_failed_so_exit_with_non_zero_val 2>nul
goto endofperl
@rem ';
#!perl
#line 15

use strict;
use warnings;

use IO::Handle   qw( );
use Getopt::Long qw( );

my $opt_a;

sub usage {
   print STDERR (<<'__EOI__');
usage: tee [<options>] [--] [file [...]]
Use tee --help for help.
__EOI__
   exit(1);
}

sub help {
   print(<<'__EOI__');
usage: tee [<options>] [--] [file [...]]
Use tee --help for help.

Options:
   -a  Append the output to the files rather than overwriting them.
   -i  Ignore the SIGINT signal.
__EOI__
   exit(0);
}

sub process_args {
   Getopt::Long::Configure('posix_default');
   Getopt::Long::GetOptions(
      "a"        => \$opt_a,
      "help|h|?" => \&help,
   ) or usage();
}

{
   process_args();

   my $mode      = ($opt_a ? '>>'   : '>'     );
   my $mode_text = ($opt_a ? 'open' : 'create');

   my @fhs;
   my %fh_names;

   {
      my $fh = \*STDOUT;
      push @fhs, $fh;
      $fh_names{$fh} = 'STDOUT';
   }
   
   foreach (@ARGV) {
      my $fh;
      if (!open($fh, $mode, $_)) {
         die("Unable to open file \"$_\": $!\n");
      }

      push @fhs, $fh;
      $fh_names{$fh} = "file \"$_\"";
   }

   binmode(STDIN);
   foreach my $fh (@fhs) {
      $fh->autoflush(1);
      binmode($fh);
   }

   for (;;) {   
      my $buf = '';
      my $rv = sysread(STDIN, $buf, 4096);
   
      if (not defined $rv) {
         die("Unable to read from STDIN: $!\n");
      }
   
      if (not $rv) {
         last;
      }
      
      foreach my $fh (@fhs) {
         if (!$fh->print($buf)) {
            die("Unable to write to $fh_names{$fh}: $!\n");
         }
      }
   }   
}

__END__
:endofperl

        [download]
[reply]
[d/l]
[select]
Re: Can't write files (even redirected from shell)
by davidrw (Prior) on Jan 14, 2008 at 04:19 UTC
    Can you post the actual code that's trying to do the file open/writes?

    (quick note: LWP::Simple::get_store is handy function too)
[reply]

      I no longer have the exact code, but it was along the lines of: 

      use IO::File;

my $file = IO::File->new(">carddb.txt") or die $!;

      [download]

      And further down, where it says print STDERR $result; it read $file->print($result) or die $!;.

      Ultimately, there was also a $file->close; at the end of the file too.

      And thank you for the reminder about LWP::Simple::get_store, but I didn't actually want to keep the whole page.

      0x596F752068617665206265656E2068657865642E

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom