Are your data files binary or text? Can you show us a small sample input file?

Update: I see by your code now that the files are text. Is it possible that they have such long lines that the read via the diamond oprator is taking a huge chunk of memory?

Update 2: Duh, I'm still asleep--please forgive my incoherent rambling about things I failed to notice in the OP.