Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Wise Monks,

How does one pipe a complex data structure (specifically an array of hashes) from a child process to a parent process? My code is able to pipe scalars from the child to the parent with no problem; furthermore, using Storage or Data::Dumper I can pipe arrays. Unfortunately, even using these modules I am unable to serialize complex data structures AND pipe them. A few notes: Storable and Data::Dumper can serialize complex data structures, but these serialized references are unable to be piped. Additionally, I have looked at Parallel::ForkManager, and it uses Storable to serialize the data so it will not work either. I am unable to find any template for how to do this.

Thank you Monks for all your help.

Scott

Replies are listed 'Best First'.
Re: Pipe Complex Data Structure
by Athanasius (Archbishop) on Sep 05, 2012 at 07:29 UTC

    I’m not sure I understand the problem — especially as you neither show the code you’ve tried nor indicate how it is failing. However...

    My code is able to pipe scalars from the child to the parent with no problem

    Well, serializing your data structure using Data::Dumper produces a string, which is a scalar, so there should be no difficulty in getting this to work with your existing pipe. But...

    these serialized references are unable to be piped

    Two possibilities come to mind:

    1. You may need to adjust the settings for $Data::Dumper::Deepcopy and/or $Data::Dumper::Purity if your structure contains cross-references.
    2. You may have a deadlock, as explained in the 2009 post Re: Issue with communication of large data between Parent and child using Perl pipes.

    I suspect that a deadlock is the more likely problem. See the referenced post for the solution (viz., run the processes asynchronously). Hope that helps,

    Athanasius <°(((><contra mundum

      Thank you Athanasius. I suspect you may be right regarding the deadlock situation; it may also be a matter of timing the writer and reader file handles correctly. Knowing that a pipe possesses a finite capacity explains a lot though.

      By the way, my code is in a post below since it was requested.

      Thank you Perl Monks for all your wisdom.

      Scott

Re: Pipe Complex Data Structure
by Anonymous Monk on Sep 05, 2012 at 07:40 UTC
Re: Pipe Complex Data Structure
by zentara (Cardinal) on Sep 05, 2012 at 10:35 UTC
Re: Pipe Complex Data Structure
by BrowserUk (Patriarch) on Sep 05, 2012 at 14:07 UTC

    See Re^3: Best technique to code/decode binary data for inter-machine communication? using a pack/unpack mechanism for the time & space efficient serialising & de-serialising of arrays and hashes.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    RIP Neil Armstrong

Re: Pipe Complex Data Structure
by Anonymous Monk on Sep 05, 2012 at 17:48 UTC

    I apologize, I did not realize the code was essential to solving the problem. The following is a sample code because the original code is much too long to list:

    # First we must create our data structure. $master_matrix->[0]->{'animal'} = "tiger"; $master_matrix->[0]->{'size'} = "big"; $master_matrix->[0]->{'speed'} = "fast"; $master_matrix->[1]->{'animal'} = "bear"; $master_matrix->[1]->{'size'} = "bigger"; $master_matrix->[1]->{'speed'} = "medium"; $master_matrix->[2]->{'animal'} = "sobes"; $master_matrix->[2]->{'size'} = "tiny"; $master_matrix->[2]->{'speed'} = "quick"; $master_array = [5, 7, 9, 12, 4]; # Now we can do the parallel processing. use IO::Handle; use Data::Dump; $max_core = 1; foreach $core_number (1..$max_core) { pipe ($reader, $writer); $writer->IO::Handle::autoflush(1); $pid = fork(); if ($pid) { # This is the parent process close $writer; undef $master_matrix; undef $master_array; $child_process_numbers{$core_number} = $pid; $child_file_handles{$core_number} = $reader; undef $reader; } elsif ($pid == 0) { # This is the child process close $reader; print "child master matrix: $#{$master_matrix}\n"; $frozen_data = Data::Dump->pp($master_matrix); print $writer $frozen_data; close $writer; exit(0); } else { # Process is undefined die "couldn't fork: $!\n"; }; }; foreach $core_number (keys %child_process_numbers) { $pid = $child_process_numbers{$core_number}; $reader = $child_file_handles{$core_number}; $frozen_data = <$reader>; undef $reader; $thawed_data = eval($frozen_data); print "thawed data: $#{$thawed_data}\n"; waitpid($pid, 0); }; exit;

    I have decided to use Data::Dump because of the Storable "version" problem (discussed on your website and several others). The code is set up to generate multiple children, but I have it set to only generate one for simplicity. The code is able to produce the correct answer if I use the $master_array, but it does not produce the correct answer if I use the $master_matrix (an array of hashes). Hopefully this will help everyone.

    Thank you Monks for all of your help.

    Scott

Re: Pipe Complex Data Structure
by Anonymous Monk on Sep 05, 2012 at 13:07 UTC
    Your serializer/unserializer code might have to be a bit more complicated, assigning unique-IDs of some sort to structures, sending them over separately and including the IDs not the structures in what you stream. Upon receipt you can rebuild the references, weakened as desired.