in reply to Running out of memory...

When you do the following... 

undef $/;

$a = <E>;

[download]
...you are reading the entire file into memory.

Try to break the task into smaller pieces. Without knowing what auburn_courses.txt or auburn_courses2.txt contain it's going to be tricky to give specific advice.

-- vek --

Replies are listed 'Best First'.
Re^2: Running out of memory...
by knewter (Novice) on Jan 29, 2005 at 20:18 UTC
    I've changed my code to the following:
    #!/usr/local/bin/perl -w
use strict;

open IN,  '< :raw', $ARGV[ 0 ] or die "$ARGV[ 0 ] : $!";
open OUT, '> :raw', $ARGV[ 1 ] or die "$ARGV[ 1 ] : $!";

my $a = '<!-- rsecftr.htm - Course Sections Table Footer -->';
my $b = '<!-- rsechdr.htm - Course Sections and Course Section Search 
+Table Header -->';

my $buffer;
sysread IN, $buffer, 5800, 5800;
do{
    ## Move the second half of the buffer to the front.
    $buffer = substr( $buffer, 5800 );

    ## and overwrite it with a new chunk
    sysread IN, $buffer, 5800, length( $buffer );

     ## Apply the regex
     $buffer =~ s|$b(.*?)$a||g;
     print $buffer;

     ## Write out the first half of the buffer
     syswrite OUT, $buffer, 5800;
} until eof IN;

close IN;
close OUT;

    [download]
    auburn_courses.txt contains a load of html files all bunched one after the other...I'd like to remove the bits between the footer of one section that I want to see and the header of the next section that I'd like to see. They're delimited by the $a and $b lines.

    Update! All fixed, ignore me
[reply]
[d/l]

      I realise you have fixed your problem, but I have to say that without seeing your actual data, 5800 is a very strange choice of buffer size?

      Examine what is said, not who speaks.
      Silence betokens consent.
      Love the truth but pardon error.
[reply]
        I don't exactly remember why I did that now, but it had something to do with the average size of the html file I was slurping...that number's something like twice it I think, so that I'm guaranteed to have the data I need in the middle somewhere...yeah, I don't remember why exactly now, but I'm glad to say that the scraper portion of my project is completed and that the data is happily in a database serving out useful information to people that didn't have it as easily before :)
[reply]

In Section Seekers of Perl Wisdom