|Don't ask to ask, just ask|
Re^3: Script exponentially slower as number of files to process increasesby marioroy (Parson)
|on Jan 28, 2023 at 05:24 UTC ( #11149966=note: print w/replies, xml )||Need Help??|
the results are very interesting...
The kikuchiyou.pl script not involving IPC should be first. It's a great solution. If memory is plentiful, why not consume a little memory for input data before spawning.
The tests ran in a loop utilizing 8, 16, 32, etc up to 4096 threads...
It's interesting seeing one attempting 4,000+ workers for a use-case that is somewhat CPU-bound. On another note, what coolness witnessing the operating system coping with this. For example, the threads and MCE::Child solutions involve IPC; e.g. workers entering a critical section -- who's next to read input from the queue or channel.
On my system, the kikuchiyo.pl script exits early running 512 workers. I modified the script to figure why that is.
Possibly a ulimit issue -n issue. My open-files ulimit is 1024. Edit: It has nothing to do with ulimit as the number of open files ulimit is per process. Workers 256+ exit early.
The threads and MCE::Child solutions pass for 512 workers. Again, regex is used here -- kinda CPU bound. Is running more workers than the number of logical CPU cores improving performance? There are golden CPU samples out there.
In Section Seekers of Perl Wisdom