Re^8: Algorithm advice sought for seaching through GB's of text (email) files

Sorry to labour the point, but having cast around for references I can't see how parallelizing an IO-bound process could ever reap performance benefits on "commodity hardware"--defined for the purposes of discussion as a single cpu system with a single harddrive.

If one process is IO-bound, then by definition, it spends most of it's time waiting for the OS kernel to complete it's IO requests. Ie. The route through the kernel IO routines, device driver and disk drive hardware is the limiting factor.

If you start a second copy of the process, then it will spend it's time waiting for it's IO requests to complete, but it will also have to wait for the IO chain to complete the first processes IO requests before it gets around to attempting to service those from the second process.

The bottleneck, wherever in the IO chain it falls, will not suddenly widen because a second process starts making requests. The only way that could happen is if the kernel held some percentage of it's potential throughput 'in reserve' for second and subsequent processes.

That's not a completely rediculous idea. Certainly some network protocols do something akin to this. SNA for example would resist allocating all the bandwidth of any given point to point link to a single end-to-end connection and would attempt to always hold some bandwidth in reserve for low-volume high priority traffic. Years ago, I remember breaking 1.4MB diskette images into smaller chunks for transmission across SNA networks as smaller transfers were always given priority over larger ones. But I've never heard, nor found reference to any dd or bus management system that does anything similar.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re^8: Algorithm advice sought for seaching through GB's of text (email) files

Replies are listed 'Best First'.
Re^9: Algorithm advice sought for seaching through GB's of text (email) files by tilly (Archbishop) on Sep 25, 2006 at 15:16 UTC
I've already told you that I have personally seen it happen. Are you implying that I am lying? That I could not have seen what I already said I did? FYI just before I went to bed last I decided to google this exact subject, and I ran across http://kerneltrap.org/node/452. The point that was made there about kernels of the same vintage as the ones that I was working with was, The current readahead model is per-inode, which is very little help with lots of small files, especially if they are fragmented or out of order. This is extremely relevant because I was working with small files of exactly that kind. If readahead works for you then latencies do not matter because the next piece of data you want has always been fetched for you before you asked for it. But if readahead does not work for you, then you will wind up waiting for the round trip time between when requests are sent to disk and the disk responds to you. So your comments about throughput are based on an assumption of a situation where the OS is able to anticipate your future requests before you make them, while my comments about latencies are based on an assumption that it is not. Your assumption is more likely to be true today. Mine certainly was true for me in the situation where I last cared to benchmark this.	[reply]
Re^10: Algorithm advice sought for seaching through GB's of text (email) files by BrowserUk (Patriarch) on Sep 25, 2006 at 16:14 UTC
Are you implying that I am lying? Dog no! I'm pursuing further information about a phenomena that I have not encountered, had not found reference to, and want to understand. I am pursuing it further simply because you are the source. Let me make that very clear. Because you are the source of the information, I have to take it seriously. That I could not have seen what I already said I did? Again no, but with respect, we have all misinterpreted imperical evidence from time to time. That's not an accusation. When I see a phenomena that doesn't gel with the logic and reason of my internal model of things, either: My model is wrong. My measurments are wrong. I'm misinterpreting the measurements. It's my habit to try and discover which. The url you posted is an interesting starting point. Thankyou. I will trouble you no further. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]