Re: Design advice: Classic boss/worker program memory consumption

Basically, you want a database that's not on the parent heap. How about this:

use GDBM_File;
tie(my %dataset, 'GDBM_File', 'tmp.db', &GDBM_NEWDB, 0600) or die;
...
[download]

I've assumed your dataset needs no serialization.

Comment on Re: Design advice: Classic boss/worker program memory consumption Download Code

Replies are listed 'Best First'.
Re^2: Design advice: Classic boss/worker program memory consumption by locked_user sundialsvc4 (Abbot) on May 21, 2014 at 17:08 UTC
A potential issue with that design is that now you have the issue of multiple read/write access to a single file ... locking, concurrency and all of that. Which could become quite messy. I know that it is somewhat counter-intuitive to have a parent that does nothing but mind the kids, but I actually find it easier to do it that way because of the issue of separation-of-concerns, or in this case, of one-thing-to-wait-on. The parent owns the kids, minds the kids, wipes their little noses, and that’s it. The kids connect to their service-provider older brother, and send results to their service-consumer older sister. Exactly how you IPC that stuff together is really up to you ... “tim toady.” But, now, each process (including the parent) basically has only one concern, one thing to worry about at any particular time, and a very uncomplicated internal structure. The processing will scale-up or scale-down easily, and be quite reliable.
Re^2: Design advice: Classic boss/worker program memory consumption by shadrack (Acolyte) on May 22, 2014 at 02:43 UTC
I've assumed your dataset needs no serialization. Unfortunately, it would. The work units are moderately complex data structures. Assembling them directly to a tied hash isn't really practical	[reply]