in reply to Out of memory problems

Read the file in chunks, unpack each chunk, check for patterns in the last chunk plus the current chunk (if the patterns may extend over chunk boundaries), pack and print half the modified data to a second file, assign the other half to last chunk, rinse and repeat. There should be no need to ever have more than a few K of data in memory.

Replies are listed 'Best First'.
Re^2: Out of memory problems
by tperdue (Sexton) on Oct 21, 2004 at 10:46 UTC
    I would have a problem with patterns extending over chunk boundaries. Do you have an example of how I would overcome this?