in reply to Use more than one threads for one file processing

Following up to hippo's reply, if the CPU usage is minimal with your current solution, then it's IO bound. In that case, there's not much you can do.

If the CPU is maxed out, then multiple threads won't help either.

It's only if you can split the file up onto multiple disks/hosts/cores, and run multiple processes, will you get much benefit. Or perhaps have a server feeding lines out to other processes on other hosts/cores, and collecting the results.

It's difficult to get around IO bound or CPU bound problems without splitting up the work between IO systems or CPU systems.

-QM
--
Quantum Mechanics: The dreams stuff is made of

  • Comment on Re: Use more than one threads for one file processing