You have two problems. The first is a misunderstanding about how Perl uses memory. When you do something that requires more memory than Perl has available, Perl grows its internal memory pool by requesting more memory from the operating system. Perl then allocates internally from this pool. As far as I know, there's no provision at present for returning memory to the operating system. A common trick in long-running Perl applications is to save state in a file, then have the application re-exec yourself.

The second problem is that you're using "slurp mode" to read the entire file at once. Unless you have a search pattern that extends across multiple lines, you can read, and search, the file line-by-line instead. The additional work this entails might be offset by the lower memory footprint it requries.

Update: If your search patterns does span lines, you might consider the technique described in Matching in huge files.

One bad thing about line-by-line in this users case though is that it will be much slower as he is reading these files over the network and the backend in windows will be way more effeciant if he pulls the whole file at once. That said if memory is more of a concern than speed line-by-line is the way to go here.

-Waswas

One bad thing about line-by-line in this users case though is that it will be much slower as he is reading these files over the network and the backend in windows will be way more effeciant if he pulls the whole file at once.

Do you have evidence to support this? My experience says the opposite. For one, reading the file in slurp mode doesn't save substantial network traffic over reading it line-at-a-time, since disk pages are read and buffered to support per-line access. For another, assuming the pattern you're trying to match occurs once and is distributed randomly through the target file, on average you'll only need to read half the file to match it.

Enter buffering. Perl doesn't read the file line by line, even if your code requests it that way.

Makeshifts last the longest.

One way from preventing the accumulation of memory that is slightly easier than having to save-state/exec/restore-state, is to spin that part of the processing that consumes large chunks off into a seperate process. When that process terminates, its memory is returned to the OS for re-use.

In your case, as you only wish to know if the file contains the string, you put a line something like this at the top of your program...

$cmd = q[ 
perl -MIO::File -we "exit( do{ local $/; my $io=new IO::File; $io->ope
+n( $ARGV[0] ); <$io> } =~ /$ARGV[1]/ ) " ];
];
[download]

If your using unix, you could probably slit that over several lines using single quotes instead of doubles.

Then replace these 4 lines...

undef $/;
my $read = new IO::File;
if($read->open("< $file")) {
if(<$read> =~ /$searchString/g) {
[download]

with...

if ( system( $cmd, $file, $searchstring) ) {
[download]

Note: You will need to adjust the $cmd string to suit your OS. There are also many ways to improve it.

For instance, you could do as dws suggested and process the file one line at a time rather than slurping it and bottle out as soon as you find the searchstring.

You'll notice that I have removed the /g option from the match. There is no point in looking for more than one occurance unless you are going to do something with the knowledge.

Also, as coded above, any failure in the script will be seen as a successful search. You should decode the return value from system, seperate the returns from perl itself from that from the exit in the one-liner. Or you could use the C-style double negative test; exit( !... ); and the  if ( !system(...) ) {.

This way, your maximum memory usage should be perl X 2 + the biggest file you process.

A good suggestion, but it's begging for some decoupling.

sub grepfile {
    my ($rx, $file) = @_;

    my $ret = system(
        qw/ perl -0777 /,
        -e => q{$a = shift; exit (<> !~ /$a/)},
        $rx, $file
    );
    return $ret == 0 if $ret != -1;

    require Carp;
    Carp::croak "Failed invoking perl for grepfile(): $!";
}
[download]

Note how using system(LIST) eliminates any quoting headaches in one easy step.

Makeshifts last the longest.

Yup! Makes it much easier. That covers most of the "many ways to improve it" I hinted at, though I'm surpised you didn't throw in the -n and exit early. Still it is much clearer haw to add that with your version.

---

Examine what is said, not who speaks.

if(<$read> =~ /$searchString/g) { 
    ...
    $read->close;
}
[download]

It isn't that Perl remembers how much memory a variable used as much as it is that Perl spent all that effort allocating it earlier, and isn't about to get rid of it just yet, as there is a high probability, in general, that if a lot of memory is needed once, it will be needed again.

sub search {

    my ($file, $searchString) = @_;

    local *F;
    open(F, "<", $file) or return 0;
    binmode(F);

    my $nbytes = read(F, my $window, 2 * BLOCKSIZE);

    while ( $nbytes > 0 ) {

        if ( $window =~ m/$searchString/ ) { close(F); return 1; }
        
        $nbytes = read(F, my $block, BLOCKSIZE);
        last if $nbytes == 0;

        substr($window, 0, BLOCKSIZE) = '';
        $window .= $block;
    }

    close(F);

    return 0;
}
[download]

if(search($file, $searchString)) { 

    print "\tFound: $file\n";

        foreach(@dir) { 

            if(/^.*?\s+([A-Z]{1}$numSet\.\S{3})$/) {

            my $found = join('\\', ($dirPath, $1));
            print "\tFound: $found\n"; 
        }
    }
}
[download]