|
Re^3: Problem in creating process
by Corion (Patriarch) on Nov 23, 2015 at 10:26 UTC
|
It could be that the machine has only one CPU available, or that the whole process is IO bound or that the program(s) are starved for RAM. "Profiling" at this high level of looking at taskmon to find out what is maxed out could at least help to optimize, or in the IO case, to stop optimizing and buy other hardware.
| [reply] |
|
|
It could be that the machine has only one CPU available, or that the whole process is IO bound or that the program(s) are starved for RAM.
I beg to differ.
Regardless of whether the machine has one or 16 cpu's, the processing as described will
- be dominated by disk head thrash (IO_bound) during the first phase (directory traversal)
- be dominated by IO during the second phase (file processing) because the simplicity of the record processing means it will take a very small amount of time relative to the time required to retrieve that record. One tenth or less.
By using a second process, the competition between the two processes during the second phase will increase the record read time; and regardless of how many cores are available do little to reduce the cpu usage required.
If 10% of clock time per record is needed for processing, the maximum gain to be had by using two processes is 5%; but the IO time will increase by far more than that due to read competition.
There really is nothing to be gained from profiling the code at this point because none of the available profilers give you any useful information about the breakdown of the IO; or the interactions between processes.
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
|
|
Ah - I hadn't looked at the actual processing code. As that code is mostly linear CSV processing and some counting, I concur with your analysis that there is little to be gained from changing the processing, as it most likely is IO.
| [reply] |
|
|
| [reply] |
Re^3: Problem in creating process
by hippo (Archbishop) on Nov 23, 2015 at 11:28 UTC
|
Given the code the OP has supplied
That's the pertinent point. The OP has not supplied the full code so we can only guess as to what the rest of it is doing. Further, the fact that the durations reported are wallclock times means that any other process could also be interfering with the resultant durations reported.
That's not to say that your suggestion that the processes are IO-bound is wrong or even unlikely and I too suspect that this will turn out to be the limiting factor in the OP's case. But it is just an guess at this point even if an educated one.
| [reply] |
|
|
any other process could also be interfering with the resultant durations reported.
But profiling (using any of the available perl profilers) won't delineate the affects of those others processors, so won't add to the information available.
I too suspect that this will turn out to be the limiting factor in the OP's case. But it is just an guess at this point even if an educated one.
I would say that:
- given the structure of the main loop;
- and that the child processes exit as soon as they have run their respective subroutines;
- And the full code of those subroutines is supplied.
Unless the OP is deliberately hiding something from us; there is simply no scope for this to be anything other than an IO bound process.
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
Re^3: Problem in creating process
by BillKSmith (Monsignor) on Nov 23, 2015 at 13:20 UTC
|
I believe that hippo is suggesting that he should have profiled the original. The parallel approach appears to be a 'guess'.
| [reply] |
|
|
I believe that hippo is suggesting that he should have profiled the original.
The OP timed both approaches; the single tasking version was faster than his attempt at multitasking. That is about as much information as you can hope to derive from profiling this kind of process.
Ie. For the single tasking approach, reading a record takes maybe 5 to 10 times longer than processing that record. No matter how much you speed up the processing code, the IO time which dominates won't change.
Ie. If the ratio between IO and processing is 10:1, the absolute best improvement he could get is a 10% saving by not performing that processing.
The parallel approach appears to be a 'guess'.
What is the difference between a "guess" and a 'trial', or an 'experiment'; or a 'test'?
And how will profiling the single tasking version inform the user such that he can avoid performing some trials, tests or experiments to improve his processing time?
The only guess here is that kneejerk response of "profile it" in response to any question that mentions "improving performance".
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |