Re^2: Keeping children alive persistiently, intelligently

Well, I almost did do this with threads... but ran into some issues. The primary issue is that some of the code called from do_childish_things() is not thread-safe.

Another issue (though I'm not *certain* it's really an issue) would have been with memory usage. From what I understand, each thread is a copy of the current interpreter's state, in the same process space as the original invocation. Due to the amount of code that makes up this app, that would be one heck of a big process.

Now, a fork() is supposed to be the same sort of thing, except that the spawned child has it's own address space... but due to the efficiency of fork() on most *nix platforms, it can use copy-on-write against the parent and thus is much more memory efficient. (Indeed, this seems to be the case from my limited inspection of the current running version)

Anyhow - there is no shared state or communication between the processes, except that they all write their output to the same file-handle (opened and initialized by the parent), using flock() to avoid stepping on each other's toes. However, all children are running the same code, and the amount of data resident in any particular child is only 10-20KB *at most*. This seems to work *very* well with fork.

I do appreciate the advice though, but I'm not convinced that using threads would solve any of the requirements I need to fill any easier.

Comment on Re^2: Keeping children alive persistiently, intelligently

Replies are listed 'Best First'.
Re^3: Keeping children alive persistiently, intelligently by weismat (Friar) on Feb 10, 2008 at 07:43 UTC
Non-thread safe code means that it can not be shared between threads, but ot does not mean that it can be used in a detached child thread. The memory usage of threaded applications can be as small as forked applications as threads can (and should be) reused if done properly. I usally work with queues with threads and detach all child threads right at the beginning. All requests and results get send then to queues. The main thread waits until the results have been received.	[reply]
Re^3: Keeping children alive persistiently, intelligently by zentara (Cardinal) on Feb 09, 2008 at 16:35 UTC
Yeah, the memory issue with threads can be bad, and forking is so clean in that respect. The way to get around it though, is to pre-make a set of empty threads right at the top of the script, and reuse them. That way, there is no overlapping useless code in the thread, and reusing them prevents memory gains. You can then load data to process into each thread thru shared variables. Yeah, it's alot of juggling to do, fork is probably better. I'm not really a human, but I play one on earth. Cogito ergo sum a bum	[reply]