in reply to Sharing large data structures between threads

An idea I have often toyed with in such situations is to daemonise the data management. So one process does nothing but insert, update, delete and fetch from the structure. The main program does an IPC::Open2 or IPC::Open3 on the data management process and the threads or forks from the main program share the filehandles to communicate with the data management process but not its memory. For example 
# data manager (short separate program)

while( <> ) {
    chomp;
    my ( $command, $value, $address ) = split /\t/;
    if ( $command eq 'insert' or $command eq 'update' ) {
        eval( "$address = $value;" );
    }
    elsif( $command eq 'delete' ) {
        delete $$address;
    }
    else{
        eval "print \$address;"
    }
}

# and the main program:

use IPC::Open2;

my $pid = open2 my $wh, my $rh, 'DataManager.pl';

...

print $wh "insert\t\"fred\"\t" . '$deepthing01{ fred }{ bert }{ pete }
+[99]';

...

[download]

One world, one people

Replies are listed 'Best First'.
Re^2: Sharing large data structures between threads
by Anonymous Monk on Mar 07, 2011 at 17:36 UTC

    Thank you for your reply!

    Your idea made me think: what if the data management was "daemonised" to a single thread instead. I would have my data structure alive in one thread only and post requests for read/write to it just as you suggested. I could even make the requests Perl code that is eval()'d by the management thread.

[reply]

      I would suggests that the requests, if you pursue that idea, “should merely be descriptive.”   If the manner by which they are (presently...) implemented is by very-clever use of eval, then so be it ... but don’t scatter that implementation detail throughout all of the client code.   One day, you might decide that this approach is not so Tom Swifty after all, and if this be so, it should be easily and (ahem...) “Swiftly” replaceable without impacting much source-code that is hither, tither and yon.

        Yes, I agree. In practice, I would create a package with a trivial new, and insert, update, delete and fetch methods as a generic interface which could be replaced by alternative solutions at a later time transparently to the caller(s).

        One world, one people

[reply]

In Section Seekers of Perl Wisdom