Re^9: Algorithm advice sought for seaching through GB's of text (email) files

I've already told you that I have personally seen it happen.

Are you implying that I am lying? That I could not have seen what I already said I did?

FYI just before I went to bed last I decided to google this exact subject, and I ran across http://kerneltrap.org/node/452. The point that was made there about kernels of the same vintage as the ones that I was working with was, The current readahead model is per-inode, which is very little help with lots of small files, especially if they are fragmented or out of order.

This is extremely relevant because I was working with small files of exactly that kind. If readahead works for you then latencies do not matter because the next piece of data you want has always been fetched for you before you asked for it. But if readahead does not work for you, then you will wind up waiting for the round trip time between when requests are sent to disk and the disk responds to you.

So your comments about throughput are based on an assumption of a situation where the OS is able to anticipate your future requests before you make them, while my comments about latencies are based on an assumption that it is not. Your assumption is more likely to be true today. Mine certainly was true for me in the situation where I last cared to benchmark this.

Comment on Re^9: Algorithm advice sought for seaching through GB's of text (email) files

Replies are listed 'Best First'.
Re^10: Algorithm advice sought for seaching through GB's of text (email) files by BrowserUk (Patriarch) on Sep 25, 2006 at 16:14 UTC
Are you implying that I am lying? Dog no! I'm pursuing further information about a phenomena that I have not encountered, had not found reference to, and want to understand. I am pursuing it further simply because you are the source. Let me make that very clear. Because you are the source of the information, I have to take it seriously. That I could not have seen what I already said I did? Again no, but with respect, we have all misinterpreted imperical evidence from time to time. That's not an accusation. When I see a phenomena that doesn't gel with the logic and reason of my internal model of things, either: My model is wrong. My measurments are wrong. I'm misinterpreting the measurements. It's my habit to try and discover which. The url you posted is an interesting starting point. Thankyou. I will trouble you no further. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]

Replies are listed 'Best First'.

Re^10: Algorithm advice sought for seaching through GB's of text (email) files
by BrowserUk (Patriarch) on Sep 25, 2006 at 16:14 UTC

Are you implying that I am lying?

Dog no! I'm pursuing further information about a phenomena that I have not encountered, had not found reference to, and want to understand. I am pursuing it further simply because you are the source.

Let me make that very clear. Because you are the source of the information, I have to take it seriously.

That I could not have seen what I already said I did?

Again no, but with respect, we have all misinterpreted imperical evidence from time to time. That's not an accusation. When I see a phenomena that doesn't gel with the logic and reason of my internal model of things, either:

My model is wrong.
My measurments are wrong.
I'm misinterpreting the measurements.

It's my habit to try and discover which.

The url you posted is an interesting starting point. Thankyou. I will trouble you no further.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

[reply]