Please show us some details. It's not possible to comment on a program you haven't shown us.

Having said that, I should add that I usually agree, when dealing with really big files, with the idea of processing it piece by piece, whether piece means a logical line or some other data block, when possible.