Re^4: Script exponentially slower as number of files to process increases

First of all, thank you for your explanations and the work you put in suggesting an alternative.

Is running more workers than the number of logical CPU cores improving performance? There are golden CPU samples out there.

I don't know what to say, other than try to benignly bypass the perlmonks' filter and show somehow the graphs (if a janitor would be kind enough to URLify these, thanks). It does seem that in certain scenarios the first part of your statement holds true.

First test, https://i.imgur.com/CpclI9L.png - 3457 files

Second test, https://i.imgur.com/cDi41fC.png 34570 files

Third test, https://i.imgur.com/yNZokCx.png - 345700 files, due to long run times, I omitted the clearly slower Threads::Queue solution.

Final test, https://i.imgur.com/2NVovHx.png - same load as in #3, but only for fork() which proved to be the fastest, across the 512-4096 range in 128-step increments, trying to find the sweet spot.

You need to copy/paste the links by hand, but it's worth the trouble. ~~The gain of fork() from 256 to 512 processes is almost unbelievable~~, while the performance of the other implementations is practically linear.

EDIT: But of course it is, it's due to workers exiting early.

I also tested your updated script but it showed no tangible improvement on my setup.

Comment on Re^4: Script exponentially slower as number of files to process increases

Replies are listed 'Best First'.
Re^5: Script exponentially slower as number of files to process increases by marioroy (Parson) on Jan 28, 2023 at 17:10 UTC
Certainly, there's an anomaly. Do you know what is causing the beautiful script to suddenly leap from 110 seconds down to 50 seconds? Unfortunately, half of the workers exit due to open file error. They go unnoticed due to no warning messages. Applying the changes here may enlighten you as to why. Another verification is running ls -l data.dat. Is the file size smaller than expected?	[reply]
Re^6: Script exponentially slower as number of files to process increases by cavac (Vicar) on Feb 02, 2023 at 13:08 UTC
I would also suggest checking the operating system logs. Depending on OPs setup, the system (or some security software) may throttle or slow down the ability to fork new processes if it thinks there is something strange going on. (Similar to how init may prevent daemons to restart too often in a given timeframe). PerlMonks XP is useless? Not anymore: XPD - Do more with your PerlMonks XP	[reply]
Re^6: Script exponentially slower as number of files to process increases by kikuchiyo (Hermit) on Jan 28, 2023 at 17:44 UTC
You could run the script with `strace -o /tmp/trace -ff -e trace=%file script.pl` to see why the file opens fail.	[reply] [d/l]
Re^7: Script exponentially slower as number of files to process increases by marioroy (Parson) on Jan 28, 2023 at 18:38 UTC
For the kikuchiyo.pl script, the fix is initializing subdircount to maxforks - 1. That ensures safety, as each worker sets $i using the same range (starting at 0). `my $subdircount = $maxforks - 1;` [download] Update: Added results for threads and MCE::Child. Running with 512 workers: Read more... (2 kB) For cpu-bound task, there's no reason to run many-many workers; beyond physical limitation. Read more... (832 Bytes)	[reply] [d/l] [select]
Re^8: Script exponentially slower as number of files to process increases by xnous (Sexton) on Jan 30, 2023 at 22:01 UTC
Re^9: Script exponentially slower as number of files to process increases by marioroy (Parson) on Jan 30, 2023 at 22:39 UTC
Re^7: Script exponentially slower as number of files to process increases by xnous (Sexton) on Jan 28, 2023 at 17:55 UTC
Because $subdir doesn't reset to 0 after the 256th fork as in the OP script, which is incidentally the $subdircount value.	[reply]
Re^6: Script exponentially slower as number of files to process increases by xnous (Sexton) on Jan 28, 2023 at 17:51 UTC
Yup, just noticed it myself and was editing my post as you posted this...	[reply]


No such thing as a small change
	PerlMonks