in reply to Re^4: write to Disk instead of RAM without using modules
in thread write to Disk instead of RAM without using modules

This is working well with some files but when I compare more files it hangs. I think it is because of memory issue.

Hangs or just runs more slowly? How many more files? What total size of files? How many records? How many bytes? How much RAM is available?

I want to modify this script so that it can work with any number of files.

So, after the second and each subsequent file is read just rip through the structure and remove any entries where the number of matches is less than the number of files read so far. That will reduce the footprint. Of course, if your original guess is wrong and this is nothing to do with memory constraints that will not help.

Update (26th of October): Beware that the stipulation in the parent that "I want to print the total count if the second line i.e the sequence matches in all files." now turns out not to be the case. Depending on the matches this now sounds like an O(N**2) problem.

  • Comment on Re^5: write to Disk instead of RAM without using modules