Out of Memory..

RobinV has asked for the wisdom of the Perl Monks concerning the following question:

Hello PerlMonks,

I am having some issues with Memory in a very simple application. That I run in Strawberry perl on Windows. The code goes like this:

    my $f1 = $ARGV[1];
    my $f2 = $ARGV[2];
    my $b1 = undef;
    my $b2 = undef;
    open f1_p, "<$f1" || die "Fatal'd\n"; 
    open f2_p, "<$f2" || die "Fatal'd\n";
      while (<f1_p>){$b1.=$_;}
      while (<f2_p>){$b2.=$_;}
    close f1_p; close f2_p;
    undef(@ARGV); #Free Memory?
    my($old, $new) = String::Diff::diff($b1, $b2);
[download]

Anywho, when the app runs it runs out of memory. This is a 4GB RAM Rig. With a 3GB SWAP File. So that's about 6GB of Memory the Application can use and run on.

The 2 files I tried are both 600MB, so in my calculation it uses. 1200MB and the extra resources that String::Dif::diff uses so thats about. 2400MB so I think I can say 3GB of RAM if I go way out of range.. But as far as I know (and correct me if I am wrong) is 3GB still smaller then 6GB so it should fit easily..

Is there an 'easy' way to fix this application?
eg setting this code to bytecode? Or anything like that? Or can I make some memory 'hacks'??

Anyway, thanks already I hope somebody can tell me where to look what to read or offer me a way to fix this.
--

Cheers,
Robin

Comment on Out of Memory.. Download Code

Replies are listed 'Best First'.
Re: Out of Memory.. by Corion (Patriarch) on Dec 15, 2009 at 08:43 UTC
Strawberry Perl is a 32bit application so it can address and use at most 4GB of memory. Your estimation of only 2.4GB required memory is optimistic - I don't see that String::Diff will not make a copy of `$b1` and `$b2`. You could try to reduce memory fragmentation and improve speed by reading the files in one go instead of reading them line by line: `$b1 = do { local $/; <$f1_p> };` [download] But I suggest you find another way to achieve what you really want. Which you haven't told us yet. For example, if you want to weed out lines that are common between the two files, a tied hash can work, as tied hashes can be stored on disk, see DB_File.	[reply] [d/l] [select]
Re^2: Out of Memory.. by ikegami (Patriarch) on Dec 15, 2009 at 15:26 UTC
`my $b1 = do { local $/; <$f1_p> };` [download] is not as good as `my $b1; { local $/; $b1 = <$f1_p> }` [download] But the OP should use `mmap` or whatever it's called on Windows.	[reply] [d/l] [select]
Re^2: Out of Memory.. by RobinV (Sexton) on Dec 15, 2009 at 09:17 UTC
Hello There, Thanks a bunch for your Quick reply. Dang Strawberry is indeed 32bit.. Dang... I need to the the big files as is as they are dumps of file systems/registry/memory and I need a Diff of 'em. So if I do Line by line I get the problem that it won't find the extra inserted lines it will see everything wrong after an insert. So its deeper then line by line. Is there a way to still do this? I ported this from C as I had a little issue with my LCS Algorithm.. -- Robin	[reply]
Re^3: Out of Memory.. by Corion (Patriarch) on Dec 15, 2009 at 09:23 UTC
I doubt that you'll get meaningful differences anyway. Briefly looking at String::Diff, I guess that it'll likely take at least quadruple the memory size of its input, because it returns strings with the differences marked. You can maybe try to port String::Diff to work with files instead of strings. Also, I'm not sure whether String::Diff has a concept of lines, and also your registry dumps likely don't have a concept of lines. Maybe using an existing, external `diff` tool is an approach. Alternatively, look at Algorithm::Diff, but I don't know whether it has more benign memory requirements.	[reply] [d/l]
Re^4: Out of Memory.. by RobinV (Sexton) on Dec 15, 2009 at 09:29 UTC