Re^4: Problem in creating process

It could be that the machine has only one CPU available, or that the whole process is IO bound or that the program(s) are starved for RAM.

I beg to differ.

Regardless of whether the machine has one or 16 cpu's, the processing as described will

be dominated by disk head thrash (IO_bound) during the first phase (directory traversal)
be dominated by IO during the second phase (file processing) because the simplicity of the record processing means it will take a very small amount of time relative to the time required to retrieve that record. One tenth or less.

By using a second process, the competition between the two processes during the second phase will increase the record read time; and regardless of how many cores are available do little to reduce the cpu usage required.

If 10% of clock time per record is needed for processing, the maximum gain to be had by using two processes is 5%; but the IO time will increase by far more than that due to read competition.

There really is nothing to be gained from profiling the code at this point because none of the available profilers give you any useful information about the breakdown of the IO; or the interactions between processes.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re^4: Problem in creating process

Replies are listed 'Best First'.
Re^5: Problem in creating process by Corion (Patriarch) on Nov 23, 2015 at 11:28 UTC
Ah - I hadn't looked at the actual processing code. As that code is mostly linear CSV processing and some counting, I concur with your analysis that there is little to be gained from changing the processing, as it most likely is IO.	[reply]