Re^6: Script exponentially slower as number of files to process increases

http://qs1969.pair.com?node_id=11149987

in reply to Re^5: Script exponentially slower as number of files to process increases
in thread Script exponentially slower as number of files to process increases

You could run the script with strace -o /tmp/trace -ff -e trace=%file script.pl to see why the file opens fail.

Comment on Re^6: Script exponentially slower as number of files to process increases Download Code

Replies are listed 'Best First'.
Re^7: Script exponentially slower as number of files to process increases by marioroy (Prior) on Jan 28, 2023 at 18:38 UTC
For the kikuchiyo.pl script, the fix is initializing subdircount to maxforks - 1. That ensures safety, as each worker sets $i using the same range (starting at 0). `my $subdircount = $maxforks - 1;` [download] Update: Added results for threads and MCE::Child. Running with 512 workers: Read more... (2 kB) For cpu-bound task, there's no reason to run many-many workers; beyond physical limitation. Read more... (832 Bytes)	[reply] [d/l] [select]
Re^8: Script exponentially slower as number of files to process increases by xnous (Sexton) on Jan 30, 2023 at 22:01 UTC
Ran some more tests, your latest mceiter2.pl script is consistently faster (albeit by a small margin) than fork.pl, however the run time includes the output file merging which is a measurable gain.	[reply]
Re^9: Script exponentially slower as number of files to process increases by marioroy (Prior) on Jan 30, 2023 at 22:39 UTC
So nice of you to write back. That script you linked to is quite efficient, considering it factors out entirely the manager process having to merge (now handled by the workers, cooperatively).	[reply]
Re^7: Script exponentially slower as number of files to process increases by xnous (Sexton) on Jan 28, 2023 at 17:55 UTC
Because $subdir doesn't reset to 0 after the 256th fork as in the OP script, which is incidentally the $subdircount value.	[reply]

In Section Seekers of Perl Wisdom