Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re: Faster and more efficient way to read a file vertically

by karlgoethebier (Abbot)
on Nov 04, 2017 at 11:14 UTC ( #1202744=note: print w/replies, xml ) Need Help??


in reply to Faster and more efficient way to read a file vertically

Stolen, cannibalized and slightly adopted from this older thread: Threads From Hell #2: How To Search A Very Huge File [SOLVED]:

#!/usr/bin/env perl # http://www.perlmonks.org/?node_id=1202693 # $Id: loop.pl,v 1.2 2017/11/04 11:02:41 karl Exp karl $ use strict; use warnings; use MCE::Loop; use Time::HiRes qw( time ); use feature qw(say); my $file = q(data.txt); MCE::Loop::init( { max_workers => 4, use_slurpio => 1 } ); my $start = time; my @result = mce_loop_f { my $slurp_ref = $_[1]; my @column; open my $fh, '<', $slurp_ref; binmode $fh, ':raw'; while (<$fh>) { push @column, substr( $_, 10, 1 ) } close $fh; MCE->gather(@column); # sleep 2; } $file; say join( '', @result ); printf "Took %.3f seconds\n", time - $start; __END__

Thanks to marioroy.

See also MCE.

Update: To avoid the call to binmode please see Encoding horridness revisited: What's going on here? [SOLVED].

Regards, Karl

«The Crux of the Biscuit is the Apostrophe»

perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1202744]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (1)
As of 2022-05-22 05:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Do you prefer to work remotely?



    Results (79 votes). Check out past polls.

    Notices?