in reply to Re: Multi threading a
in thread Multi threading a

Hello wiser choroba,

> Reading from a single file in multiple threads tends to be slower

Are you sure? Is not what MCE::Grep is for?

## File path, glob ref, IO::All::{ File, Pipe, STDIO } obj, or scalar 
+ref
## Workers read directly and not involve the manager process
my @e = mce_grep_f { /pattern/ } "/path/to/file"; # efficient

[download]

Also: MCE::Grep#PARSING-HUGE-FILES

L*

There are no rules, there are no thumbs..
Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.

Replies are listed 'Best First'.
Re^3: Multi threading a.. huge file -- MCE::Grep
by choroba (Cardinal) on Sep 03, 2021 at 07:55 UTC
    Oh, so in each thread, you read a large chunk from the file and then process it line by line in memory! This might work if you're sure your pattern can't be split between two chunks.

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
[reply]
[d/l]
Re^3: Multi threading a.. huge file -- MCE::Grep
by karlgoethebier (Abbot) on Sep 03, 2021 at 17:33 UTC
[reply]

In Section Seekers of Perl Wisdom