Re^2: use threads for dir tree walking really hurts

Thank you very much for your vivid elaboration which is very inspiring. =D

Why are you doing that?

Because when an object fall into a thread scope, the object will be cloned, which is not the one I want. And since threads don't share object / complex data structure ( and I don't what to share them one by one ), this trick do share the object perfectly... until it's not.

Actually, I can do the job with simply: @dirToFetch : shared, but same issue Thread::Queue, I gotta leave it at a nested package scope , but create it inside an object become another mess to share around threads. Because I attempt to make it a module, so I hope to avoid if other script calling this module in threads, the data will mess up.

Though, I've update my OP's code, which will work and as fast as dir /s/b I create as many threads as how much in @dirToFetch

Comment on Re^2: use threads for dir tree walking really hurts Select or Download Code

Replies are listed 'Best First'.
Re^3: use threads for dir tree walking really hurts by Corion (Patriarch) on Sep 01, 2016 at 14:08 UTC
Why are you doing that? Because when an object fall into a thread scope, the object will be cloned, which is not the one I want. And since threads don't share object / complex data structure ( and I don't what to share them one by one ), this trick do share the object perfectly... until it's not. Yes - due to Perls reference counting, accessing variables in another threads memory always means that your thread will also be writing at least to the refcount field of that variable. If the refcount happens to reach zero in another thread than where the piece of memory was originally allocated, the memory will be freed in the wrong thread context, which is not fun. I'm not aware of a way to make Perl skip its refcounting for variables, and I'm also not convinced that this could work except in the most trivial cases. Another idea to reduce the conceptual load of the appriach might be to simply shell out to `cmd /c "dir /b /s $directory"`, but then you need to be aware of the codepage that `cmd.exe` uses for its output. Ideally you have set the codepage to Unicode / 65001: `chcp 65001` [download] ... but then, you still have to live with the fun of Perl and the OS treating the octets for filenames differently unless you properly decode and encode them.	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^3: use threads for dir tree walking really hurts
by Corion (Patriarch) on Sep 01, 2016 at 14:08 UTC

Why are you doing that?
Because when an object fall into a thread scope, the object will be cloned, which is not the one I want. And since threads don't share object / complex data structure ( and I don't what to share them one by one ), this trick do share the object perfectly... until it's not.

Yes - due to Perls reference counting, accessing variables in another threads memory always means that your thread will also be writing at least to the refcount field of that variable. If the refcount happens to reach zero in another thread than where the piece of memory was originally allocated, the memory will be freed in the wrong thread context, which is not fun.

I'm not aware of a way to make Perl skip its refcounting for variables, and I'm also not convinced that this could work except in the most trivial cases.

Another idea to reduce the conceptual load of the appriach might be to simply shell out to cmd /c "dir /b /s $directory", but then you need to be aware of the codepage that cmd.exe uses for its output. Ideally you have set the codepage to Unicode / 65001:

chcp 65001
[download]

... but then, you still have to live with the fun of Perl and the OS treating the octets for filenames differently unless you properly decode and encode them.

[reply]
[d/l]
[select]