Re^6: Perl threading stability?

in reply to Re^5: Perl threading stability?
in thread Perl threading stability?

Not only does it sound doable, it sounds pretty straight forward.

I'm pretty sure it is pretty straight forward. My only concern is taking up too much resources. The current script (using max if 5 fork()'d children) already takes ~4-5 hours to run... Albeit some things aren't at it's most efficient, I just don't want it to take longer.

From what I'm seeing with Perl threads is that it's really CPU/Memory intensive vs general fork(). That's a lot to ask for just for the ability to share data, my primary concern. Not to mention Perl's history of being flakey with Threads (although a post below has stated it's stablalized now).

In the future I might see about writing this in Java, but at this time it's just not feasiable due to current data imports (standard Dumper dump of variables into a file). But that's not for at least 6+ months, until I can get XML inputs and data use stablized.

If you want to chat more about specs, feel free to email me gp-at-gpcentre.net. I don't really want to clutter this post up with "my problem" per say. More use it to discusse perl threading in general.

-- philip
We put the 'K' in kwality!

Comment on Re^6: Perl threading stability?

Replies are listed 'Best First'.
Re^7: Perl threading stability? by BrowserUk (Patriarch) on Jul 26, 2005 at 21:27 UTC
My only concern is taking up too much resources. With appropriate care, spawning a thread can take as little as 300/400k. It's just a case of ensuring that each thread only loads what it needs. Your main concern (reading between the lines still) is that sharing a large hash between many threads will cause large amounts of duplication--and your right, if you allow the default behaviour to take it's course, it will--but there are some fairly simple techniques for getting around that in most cases. It really comes down to only sharing what is required a given thread with that thread. Whilst your overall application may have need to retain a large volume of data in memory, for most applications, each thread only needs access to some small part of the overall dataset, at any given time. As for the spec, I'm really interested in seeing how iThreads can be used. I'll drop you a note and we can move on from there. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.	[reply]

Replies are listed 'Best First'.

Re^7: Perl threading stability?
by BrowserUk (Patriarch) on Jul 26, 2005 at 21:27 UTC

My only concern is taking up too much resources.

With appropriate care, spawning a thread can take as little as 300/400k. It's just a case of ensuring that each thread only loads what it needs. Your main concern (reading between the lines still) is that sharing a large hash between many threads will cause large amounts of duplication--and your right, if you allow the default behaviour to take it's course, it will--but there are some fairly simple techniques for getting around that in most cases.

It really comes down to only sharing what is required a given thread with that thread. Whilst your overall application may have need to retain a large volume of data in memory, for most applications, each thread only needs access to some small part of the overall dataset, at any given time.

As for the spec, I'm really interested in seeing how iThreads can be used. I'll drop you a note and we can move on from there.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

[reply]

In Section Seekers of Perl Wisdom