Could you post (a link to) your latest code?

It isn't nearly finished and is undergoing a transformation since I typed my response, but at the time of this writing you can find the threading code here

One of the first things you'll notice is that I didn't implement a "guard" or "doorman". I'm just stuffing everything in the queue for now. Even on directory trees larger than 200GB it barely takes more than 200MB RAM.

Tommy
A mistake can be valuable or costly, depending on how faithfully you pursue correction

I'd suggest trying the following modification:

my $thread_term   :shared = 0;
my $threads_init  :shared = 0;
my $file_read_sem : shared;                                           
+                   ### Add this

...

   WORKER: while ( !$thread_term && defined ( my $file = $work_queue->
+dequeue ) )
   {
     {                                                                
+                   ### Add this
         lock $file_read_sem;                                         
+                   ### Add this
         open my $fh, '<', $file or do { lock $d_counter; $d_counter++
+; next WORKER };

         my $data = <$fh>;

         close $fh;
      }                                                               
+                   ### Add this
[download]

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

It isn't nearly finished and is undergoing a transformation since I typed my response, but at the time of this writing you can find the threading code here

One of the first things you'll notice is that I didn't implement a "guard" or "doorman". I'm just stuffing everything in the queue for now. Even on directory trees larger than 200GB it barely takes more than 200MB RAM.

Tommy
A mistake can be valuable or costly, depending on how faithfully you pursue correction