|
|
|
| We don't bite newbies here... much | |
| PerlMonks |
comment on |
| ( [id://3333]=superdoc: print w/replies, xml ) | Need Help?? |
|
I never got around to the blog post I'd intended for the above code- some commentary:
I doubt that more CPU can fetch a physical hard drive faster than a single one.When in doubt, benchmark! Both Mario & I timed a variety of ways to walk a directory tree, and well-written multi-threaded file walkers generally beat out the single-threaded ones. I'm not able to dig up my variants now, but if you're curious, compare what Mario posted, and the OP's working code, with File::Find on a few directory trees on the systems of your choice. My goal was to divide up the work with as little overhead as possible. And I wanted it general enough so that all threads would be busy, regardless of working in 1 directory with a million files, or a directory tree with a million leaves and no files. Above is the result of about eight different methods, with Mario's help- and there are still a couple more ideas I'd like to benchmark, whenever I can get back to it- I think there are still more efficient ways! One thing benchmarking shows, is that even though multi-threaded walkers can finish faster- they can use less wallclock time- CPU time always increases: it simply takes more total cycles, when you add up the work from all CPUs. Be careful of what the Benchmark module is really telling you! (Thanks Mario for making my code look better than the original & tweaking it too.) In reply to Re^2: use threads for dir tree walking really hurts
by Yary
|
|