Re: Threads slurping a directory and processing before conclusion

 while ( my $bl = $$q->dequeue_nb()){
[download]

There is almost no reason to ever use ->dequeue_nb() in a worker thread. Enqueue undef to tell a worker thread to quit, and enqueue a value (or an arrayref of values for bulk processing) when the worker should do something.

sub slurp_directory_as_bin{
...
[download]

Personally, if you have that many files, I would consider to either store all the file names in a database, or to spawn an external process (like ls -lR or dir /b /s) to read in all the filenames. With the database you get more flexibility to slice and dice your dataset, and if it doesn't change (that often), having convenient access makes all the other things more convenient too. With the external process, you can start processing the first files in the "directory reader thread" while it still reads in more files.

Comment on Re: Threads slurping a directory and processing before conclusion Select or Download Code

Replies are listed 'Best First'.
Re^2: Threads slurping a directory and processing before conclusion by TRoderic (Novice) on Aug 26, 2011 at 22:06 UTC
I'd never seen any documentation mentioned that enqueue undef did that, thanks for highlighting it.	[reply]
Re^3: Threads slurping a directory and processing before conclusion by Corion (Patriarch) on Aug 27, 2011 at 07:20 UTC
There is no documentation mentioning that because you have to do all of this yourself. You will need to make the receiving threads quit once they read a `undef` from their input queue.	[reply] [d/l]