in reply to Re^8: Threads From Hell #2: How To Parse A Very Huge File
in thread Threads From Hell #2: How To Search A Very Huge File [SOLVED]
MCE has the parallel_io option which is not enabled by default. This seems like a good time to try it. Btw, I do not recommend enabling this option if reading from a NFS server or from mechanical drives. IO reads for an input file are normally sequential with only one worker reading at a time.
MCE::Loop::init { max_workers => 4, parallel_io => 1, use_slurpio => 1; }; $ time ./mce_loop_script.pl nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl nose cuke karl Took 23.541 seconds real 0m23.670s user 0m51.046s sys 0m 7.150s
That chunks at 0.85 GB per second and not able to go any faster.
|
---|