in reply to Re: How to improve speed of reading big files
in thread How to improve speed of reading big files

I like graff's benchmark idea. I'd like to point out something that hopefully will happen.... if you run the benchmark the second time, it may speed up!!

I don't know what OS the OP has or how big total file size is, but there is a level of caching that the OS does that Perl doesn't even see.

As an example, I have one app that works with a dataset of 700 text files. The user can use a search window to find various things in this dataset. The first search takes about 7 seconds. The second search is so fast that it is barely perceptible to the user (>10x performance). The reason why things get faster is that my machine has lots of memory that the OS can use for file cache. The OS knows that these files haven't been modified and there is very little disk access going on. I am running on WinXP which isn't exactly famous for performance or "smarts".

The number of things that can affect I/O performance are legion and I have very little information to go on in the OP's case. Anyway in this particular app, I quit optimizing because on user query #8, they have already forgotten that query #1 took a long longer!

  • Comment on Re^2: How to improve speed of reading big files