in reply to Re: Reading from file in threaded code is slow
in thread Reading from file in threaded code is slow

Firstly, my production code is CPU bound; the example above is not. My single threaded production code on my 4 core box runs at 100% of one core.

Really? Cos when I run a single process reading 10 x 4MB files each consisting of 16,000 lines on my 4-core system, it reads and counts all 40MB/160,000 lines in ~1/3 of a second:

c:\test>dir junk?.dat Volume in drive C has no label. Volume Serial Number is 8C78-4B42 Directory of c:\test 09/07/2011 01:39 4,032,000 junk0.dat 09/07/2011 01:39 4,032,000 junk1.dat 09/07/2011 01:39 4,032,000 junk2.dat 09/07/2011 01:39 4,032,000 junk3.dat 09/07/2011 01:39 4,032,000 junk4.dat 09/07/2011 01:39 4,032,000 junk5.dat 09/07/2011 01:39 4,032,000 junk6.dat 09/07/2011 01:39 4,032,000 junk7.dat 09/07/2011 01:39 4,032,000 junk8.dat 09/07/2011 01:39 4,032,000 junk9.dat 10 File(s) 40,320,000 bytes 0 Dir(s) 205,414,293,504 bytes free c:\test>head junk0.dat 1234567890123456789012345678901234567890123456789012345678901234567890 +123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456789012345678901234567890123456789012345678 +901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 +123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456789012345678901234567890123456789012345678 +901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 +123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456789012345678901234567890123456789012345678 +901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 +123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456789012345678901234567890123456789012345678 +901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 +123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456789012345678901234567890123456789012345678 +901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 +123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456789012345678901234567890123456789012345678 +901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 +123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456789012345678901234567890123456789012345678 +901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 +123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456789012345678901234567890123456789012345678 +901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 +123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456789012345678901234567890123456789012345678 +901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 +123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456789012345678901234567890123456789012345678 +901234567890123456789012345678901234567890 c:\test>wc -l junk0.dat 16000 junk0.dat c:\test>perl -MTime::HiRes=time -e"BEGIN{ @ARGV = map glob, @ARGV; print time}" -nle"$n++ }{ print $n, ' ', time" junk?.dat 1310172599.189 160000 1310172599.43053

And the cpu usage is 2.7% for less than 1/2 a second. It requires more than that to start a single thread.

So, if you are going to make a case for threading this insignificant piece of code, you are going to have to show something far more convincing than you have so far. Seeing is believing.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.