Re^8: Threaded Code Not Faster Than Non-Threaded -- Why?

It isn't nearly finished and is undergoing a transformation since I typed my response, but at the time of this writing you can find the threading code here

One of the first things you'll notice is that I didn't implement a "guard" or "doorman". I'm just stuffing everything in the queue for now. Even on directory trees larger than 200GB it barely takes more than 200MB RAM.

Tommy
A mistake can be valuable or costly, depending on how faithfully you pursue correction

Comment on Re^8: Threaded Code Not Faster Than Non-Threaded -- Why?

Replies are listed 'Best First'.
Re^9: Threaded Code Not Faster Than Non-Threaded -- Why? by BrowserUk (Patriarch) on Jan 06, 2014 at 07:06 UTC
I'd suggest trying the following modification: `my $thread_term :shared = 0; my $threads_init :shared = 0; my $file_read_sem : shared; + ### Add this ... WORKER: while ( !$thread_term && defined ( my $file = $work_queue-> +dequeue ) ) { { + ### Add this lock $file_read_sem; + ### Add this open my $fh, '<', $file or do { lock $d_counter; $d_counter++ +; next WORKER }; my $data = <$fh>; close $fh; } + ### Add this` [download] With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]
Re^10: Threaded Code Not Faster Than Non-Threaded -- Why? by Tommy (Chaplain) on Jan 06, 2014 at 18:27 UTC
I'd suggest trying the following modification ...To prevent disk thrashing? Tommy A mistake can be valuable or costly, depending on how faithfully you pursue correction	[reply]
Re^11: Threaded Code Not Faster Than Non-Threaded -- Why? by BrowserUk (Patriarch) on Jan 06, 2014 at 19:57 UTC
To prevent disk thrashing? Yes. But you'll need to try it; and the effects will probably vary for different file systems. For lots of small files it might speed things up; for lots of big ones it might slow things down. For a mix; you'll need to measure. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]