in reply to Re: optimization in file processing
in thread optimization in file processing

While your consideration is very valuable, I think it also shows that the OP's speed problem likely doesn't stem from IO, at least if his machine behaves similarly as yours.

10 lakhs = 1 million, so to be IO bound his record size would need be in the order of 1 MB. I don't know what data the OP deals with, but typical CSV files have much smaller records.

Replies are listed 'Best First'.
Re^3: optimization in file processing
by jethro (Monsignor) on Jul 08, 2011 at 11:57 UTC
    He said his file size is in the terabyte range (at least that's what I think his last sentence said). So either he has an atypical CSV file or one of his numbers is wrong.