RE: Re: read X number of lines?

wow... i changed the read size to 8192 (2 4k blocks...) and I got these results on a 16 meg "test" file:

Benchmark: timing 10 iterations of chunk, linebyline...
     chunk:  9 wallclock secs ( 7.66 usr +  1.58 sys =  9.24 CPU)
linebyline: 34 wallclock secs (33.17 usr +  0.80 sys = 33.97 CPU)
[download]

damn... that makes a BIG difference... screw line by line...

Comment on RE: Re: read X number of lines? Download Code

Replies are listed 'Best First'.
RE: RE: Re: read X number of lines? by takshaka (Friar) on May 26, 2000 at 10:59 UTC
You can even squeeze a little more out. This is ~5% faster on my system. `my($buf, $chunk, @lines); while(read FILE, $buf, BUFFER_SIZE) { $chunk .= $buf; @lines = split /\n/, $chunk; $chunk = chomp $buf ? '' : pop @lines; foreach (@lines) {} }` [download] (BUFFER_SIZE is just a constant I was using)	[reply] [d/l]
RE: RE: Re: read X number of lines? by Anonymous Monk on May 26, 2000 at 11:33 UTC
Although this is a Perl site, when you start playing with things as a block of data, I start thinking C/C++. Perhaps you should consider that as well... For data organized in \t and \n delimiters you should be able to parse it quickly in C. Whether you can do whatever operations on that data easily in C is another question, but one I think is worth asking yourself if runtime is a serious consideration. I'd also double check for anyplace you might need an eof(). Possible corner cases that might be good tests for the chunk() code is where you have a \n as last char in file, and where you do not.	[reply]