Re: Keeping children alive persistiently, intelligently

Fellow friar, frequent flights of fancy flip my fallow folds. But I would consider using threads. Why? It seems you need alot of interprocess communication, and threads make it easy. You can setup alot of shared variables and a loop or timer mechanism to track them. You can have a set of shared variables in each thread to store it's current parameters, so you can restart it. You can reuse threads too. In a threaded app, if any thread runs "exit" it will kill all threads.

But exactly how you setup your threads depends on things like what are they doing, do you want to reuse them, how many simultaneously running threads do you want? SuperSearch here for threads, or groups.google.com will yield many examples.

Generally I like to run threads and have the main parent thread just act as a controller( watching, reading shared vars, etc). It is convenient to do it with an event-loop system like POE, Tk, Gtk2, or even GLib (my current commandline favorite).

I'm not really a human, but I play one on earth. Cogito ergo sum a bum

Comment on Re: Keeping children alive persistiently, intelligently

Replies are listed 'Best First'.
Re^2: Keeping children alive persistiently, intelligently by Hercynium (Hermit) on Feb 08, 2008 at 20:26 UTC
Well, I almost did do this with threads... but ran into some issues. The primary issue is that some of the code called from do_childish_things() is not thread-safe. Another issue (though I'm not certain it's really an issue) would have been with memory usage. From what I understand, each thread is a copy of the current interpreter's state, in the same process space as the original invocation. Due to the amount of code that makes up this app, that would be one heck of a big process. Now, a fork() is supposed to be the same sort of thing, except that the spawned child has it's own address space... but due to the efficiency of fork() on most nix platforms, it can use copy-on-write against the parent and thus is much more memory efficient. (Indeed, this seems to be the case from my limited inspection of the current running version) Anyhow - there is no shared state or communication between the processes, except that they all write their output to the same file-handle (opened and initialized by the parent), using flock() to avoid stepping on each other's toes. However, all children are running the same code, and the amount of data resident in any particular child is only 10-20KB at most. This seems to work very* well with fork. I do appreciate the advice though, but I'm not convinced that using threads would solve any of the requirements I need to fill any easier.	[reply]
Re^3: Keeping children alive persistiently, intelligently by weismat (Friar) on Feb 10, 2008 at 07:43 UTC
Non-thread safe code means that it can not be shared between threads, but ot does not mean that it can be used in a detached child thread. The memory usage of threaded applications can be as small as forked applications as threads can (and should be) reused if done properly. I usally work with queues with threads and detach all child threads right at the beginning. All requests and results get send then to queues. The main thread waits until the results have been received.	[reply]
Re^3: Keeping children alive persistiently, intelligently by zentara (Cardinal) on Feb 09, 2008 at 16:35 UTC
Yeah, the memory issue with threads can be bad, and forking is so clean in that respect. The way to get around it though, is to pre-make a set of empty threads right at the top of the script, and reuse them. That way, there is no overlapping useless code in the thread, and reusing them prevents memory gains. You can then load data to process into each thread thru shared variables. Yeah, it's alot of juggling to do, fork is probably better. I'm not really a human, but I play one on earth. Cogito ergo sum a bum	[reply]