http://qs1969.pair.com?node_id=326108

xiper has asked for the wisdom of the Perl Monks concerning the following question:

I'm looking for a way to share data, in a non-blocking, asynchronous type of way, between parent & child (& further-child) on win32. The data is a complex data structure, so i'd rather avoid manually serialising it and pumping it through a pipe or socket. I'd also like to avoid tieing & untieing to a file over and over, for obvious reasons... A tied shared variable, something like IPC::Shareable, would be nice, but it requires System V IPC type os calls. I've tried Tie::Win32MemMap (recommended by Mastering Perl/Tk, but nowhere to be found on CPAN), but it appears to be quiet buggy during my tests. Win32::Semaphore has been suggested, although it only shares a single number... any other suggestions?

Update: Some background, for those wanting to solve the 'bigger picture':

I have a Tk front-end that needs make calls to a linux back-end server over tcp, for which i am using SOAP. No problems there. However, SOAP blocks while making a call, which of course will stall the Tk event-loop. So the front-end forks (before any Tk stuff) a 'handler', whose job it is to do any time-consuming background processing, such as SOAP calls, while the event-loop cycles.

My plan was, and i already have the test code to do this, to have the Tk process pipe a command to the 'handler' and waitVariable( \%share{wait} ) on a shared hash, with args in $share{in}. The 'handler' does SOAPy-type stuff (in another sub-process so it can be canceled/killed/timed-out, if needed), writes the return data to $share{out} when finished, and $share{wait}++'s to continue the Tk code.

This method has the following advantages:

  • Doesn't stall the Tk event-loop
  • Can pass complex data structures both directions easily
  • Allows timeouts
  • Allows the user to click Cancel to abort processing
  • Easily portable
  • The Tk code only has to say %data = dispatch( 'get_data', $user ); or dispatch( 'cancel' ); and everything else is transparent

    Unfortunately, i was relying on Tie::Win32MemMap to achieve the win32 side of this (this app will also run locally on the linux box). On testing, the underlying module Win32::MemMap (whose author seems to have gone AWOL), appears to be built for WinNT, and fails spectacularly in an alarming variety of ways under XP...

    Code is available if required, in case my description wasn't that clear.

  • - ><iper

    use japh; print;

    Replies are listed 'Best First'.
    Re: Shared variables under win32?
    by BrowserUk (Patriarch) on Feb 03, 2004 at 05:26 UTC
        The suggestion to use the clipboard as an IPC method is a pretty sad commentary. The clipboard is a user interface concept, not another anonymous pipe you can open between two processes.

        Scenario: The user starts your scripts, then goes on to do something entirely different, and on the first Ctrl+C, hoses your scripts. Of course, it's not obvious that the scripts have failed, and it probably won't be obvious why they failed. Unless the user does a Ctrl+V at some point, and chunks of perl object data are spewed into their document.

        From an architectural point of view, it's merely a lack of thought about the data flow: other processes may poison or be poisoned by your use of that particular IPC channel. From a user design point of view, it's damaging the user interaction model by subverting the user's metaphor to grasp of what's going on in the computer.

        Poking stuff into the clipboard is pretty cool: when the user directs the activity. Grabbing stuff from the clipboard is pretty cool: when the user directs the activity. The APIs were made quick and simple, to facilitate application designers who wanted to make their apps quick and simple.

        --
        [ e d @ h a l l e y . c c ]

        Win32::Clipboard can only write text to the clipboard (doesn't rule out Data::Dumper or MIME::Base64, although the concept still sounds messy). Will investigate Win32::API::Prototype - any examples? It's been a while since i played with the win32 api...

        - ><iper

        use japh; print;
    Re: Shared variables under win32?
    by Roger (Parson) on Feb 03, 2004 at 05:27 UTC
      You could try the Cache::Cache package, also available at the ActiveState ppm repository in binary form.

      The module in particular is called Cache::SharedMemoryCache, which looks pretty good to me.

      (I am doing experiments with this module at the moment...)

        Sounds good, although Cache::SharedMemoryCache requires IPC::ShareLite which requires ... you guessed it!

        - ><iper

        use japh; print;
    Re: Shared variables under win32?
    by flyingmoose (Priest) on Feb 03, 2004 at 14:24 UTC
      "The data is a complex data structure, so i'd rather avoid manually serialising it and pumping it through a pipe or socket"

      Does Data::Dumper or the YAML module count as "manual serialization"? They are pretty painless to me, and work great for this sort of thing.

        The sentence you quoted actually meant to emphasise the latter half, not the former.

        The data is actually serialisable using the methods you mention, the issue is transferring that data between processes 'asynchronously, without blocking' (for lack of a better way to describe it - iíve just added a background <readmore> to the original post).

        - ><iper

        use japh; print;
    Re: Shared variables under win32?
    by NetWallah (Canon) on Feb 03, 2004 at 05:46 UTC
      My conclusions are about the same as yours - it looks like "roll your own" time.

      The IPC::Sharable module basically serializes the shared variable, transports it, then re-creates it (with fancy locking and 2-way Ties). I don't see any way to avoid the serialization/re-assembly.

      You could potentially re-design your app such that instead of exposing the data, you expose methods to manipulate the data. Then your problem comes down to (relatively) simple RPC.

      "When you are faced with a dilemma, might as well make dilemmanade. "
    Re: Shared variables under win32?
    by ysth (Canon) on Feb 03, 2004 at 16:53 UTC
      Probably won't help you, but if using cygwin perl is an option, you can (with some work) get SysV IPC working there. I'm hoping it will work more out-of-the-box with perl5.8.4 (and cygwin 1.5.7).
    Re: Shared variables under win32?
    by bobtfish (Scribe) on Feb 04, 2004 at 12:53 UTC
      Can you let me know what you decide to do with this problem, as it is very similar to my question here
    Re: Shared variables under win32?
    by Roger (Parson) on Feb 06, 2004 at 16:51 UTC
      I have written a proper Win32:MMF module for shared memory support under Windows. I have tested it under .Net and VC 6. It works under Windows NT, 2000 and XP. I haven't tested it under 98, but who cares about 98's anyway?

      The following is a snip on using the module mentioned -
      use strict; use warnings; use Win32::MMF; use Data::Dumper; use CGI; # fork a process defined(my $pid = fork()) or die "Can not fork a child process!"; if ($pid) { my $ns1 = Win32::MMF->new ( -namespace => "My.data1" ); my $ns2 = Win32::MMF->new ( -namespace => "My.data2" ); my $cgi = new CGI; my $data = {a=>[1,2,3], b=>4, c=>"A\0B\0C\0"}; $ns1->write($data); # automatic locking by default $ns2->write($cgi); print "--- Sent ---\n"; print Dumper($data), "\n"; print Dumper($cgi), "\n"; sleep(1); } else { # in child sleep(1); my $ns1 = Win32::MMF->new ( -namespace => "My.data1", -reuse => 1 ) or die "Namespace does not exist!"; my $ns2 = Win32::MMF->new ( -namespace => "My.data2", -reuse => 1 ) or die "Namespace does not exist!"; my $data = $ns1->read(); my $cgi = $ns2->read(); print "--- Received ---\n"; print Dumper($data), "\n"; print Dumper($cgi), "\n"; print "--- Use Received Object ---\n"; # use the object from another process :-) print $cgi->header(), $cgi->start_html(), "\n", $cgi->end_html(), "\n"; }

      The above is just a simple demo on the module. To impliment something like what you have suggested -
      # turn off built-in auto-locking my $ns = Win32::MMF->new ( -namespace => "My.data1", -size => 2*1024*1024, -autolock => 0 ); - use Win32::Semaphore module - Create a read and a write semaphore. - In server: wait($read_semaphore); $xml = $ns->read(); release($write_semaphore); - In client: wait($write_semaphore); $ns->write($xml); release($read_semaphore);

      You can also pass blessed objects from one process into another and pick up execution in another process.