Re: Out of Memory..

Strawberry Perl is a 32bit application so it can address and use at most 4GB of memory.

Your estimation of only 2.4GB required memory is optimistic - I don't see that String::Diff will not make a copy of $b1 and $b2.

You could try to reduce memory fragmentation and improve speed by reading the files in one go instead of reading them line by line:

$b1 = do { local $/; <$f1_p> };
[download]

But I suggest you find another way to achieve what you really want. Which you haven't told us yet. For example, if you want to weed out lines that are common between the two files, a tied hash can work, as tied hashes can be stored on disk, see DB_File.

Comment on Re: Out of Memory.. Select or Download Code

Replies are listed 'Best First'.
Re^2: Out of Memory.. by ikegami (Patriarch) on Dec 15, 2009 at 15:26 UTC
`my $b1 = do { local $/; <$f1_p> };` [download] is not as good as `my $b1; { local $/; $b1 = <$f1_p> }` [download] But the OP should use `mmap` or whatever it's called on Windows.	[reply] [d/l] [select]
Re^2: Out of Memory.. by RobinV (Sexton) on Dec 15, 2009 at 09:17 UTC
Hello There, Thanks a bunch for your Quick reply. Dang Strawberry is indeed 32bit.. Dang... I need to the the big files as is as they are dumps of file systems/registry/memory and I need a Diff of 'em. So if I do Line by line I get the problem that it won't find the extra inserted lines it will see everything wrong after an insert. So its deeper then line by line. Is there a way to still do this? I ported this from C as I had a little issue with my LCS Algorithm.. -- Robin	[reply]
Re^3: Out of Memory.. by Corion (Patriarch) on Dec 15, 2009 at 09:23 UTC
I doubt that you'll get meaningful differences anyway. Briefly looking at String::Diff, I guess that it'll likely take at least quadruple the memory size of its input, because it returns strings with the differences marked. You can maybe try to port String::Diff to work with files instead of strings. Also, I'm not sure whether String::Diff has a concept of lines, and also your registry dumps likely don't have a concept of lines. Maybe using an existing, external `diff` tool is an approach. Alternatively, look at Algorithm::Diff, but I don't know whether it has more benign memory requirements.	[reply] [d/l]
Re^4: Out of Memory.. by RobinV (Sexton) on Dec 15, 2009 at 09:29 UTC
Thanks I'll have a look at Algorithm::Diff. Also, I found the BasicPerl Function Tie I never saw, I ain't sure (havn't read exactly what it does yet) but in the Sample Codes it looks usefull. Thanks anyway, Ill have a look into these things. Ill rewrite this. And by the end of the day if I couldn't find a way to fix this I'll have to breakout the nasty qx!!; to execute an external command. Thanks! -- Robin	[reply]