Beefy Boxes and Bandwidth Generously Provided by pair Networks
We don't bite newbies here... much
 
PerlMonks  

(Updated Again) Win32::MMF - Memory Mapped File Support for Perl

by Roger (Parson)
on Feb 05, 2004 at 16:21 UTC ( [id://326785]=CUFP: print w/replies, xml ) Need Help??

I have just uploaded the Win32::MMF module to PAUSE, which will (hopefully) appear in CPAN soon. Big thanks to PodMaster for picking the namespace for me. And thanks to NetWallah for testing and ideas on OO :-)

This module is primarily designed for inter-process and intra-process communication under Windows, using the Windows own native memory mapped file facility - the backbone of Windows virtual memory.

The core of the module is written in XS for performance. This is the first release of the module so the functionalities are limited. The following is a demo on how to do intra/inter-process communication using the module. Any suggestions and beta-testing are greatly welcome, especially on locking. :-)

use strict; use warnings; use Win32::MMF; use Data::Dumper; use CGI; # for testing of inter-process object transportation # fork a process defined(my $pid = fork()) or die "Can not fork a child process!"; if ($pid) { my $ns1 = Win32::MMF->new ( -namespace => "My.data1" ); my $ns2 = Win32::MMF->new ( -namespace => "My.data2" ); my $cgi = new CGI; my $data = {a=>[1,2,3], b=>4, c=>"A\0B\0C\0"}; $ns1->write($data); # autolocking by default $ns2->write($cgi); print "--- Sent ---\n"; print Dumper($data), "\n"; print Dumper($cgi), "\n"; sleep(1); } else { # in child sleep(1); my $ns1 = Win32::MMF->new ( -namespace => "My.data1", -nocreate => 1 ) or die "Namespace does not exist!"; my $ns2 = Win32::MMF->new ( -namespace => "My.data2", -nocreate => 1 ) or die "Namespace does not exist!"; my $data = $ns1->read(); my $cgi = $ns2->read(); print "--- Received ---\n"; print Dumper($data), "\n"; print Dumper($cgi), "\n"; print "--- Use Received Object ---\n"; # use the object from another process :-) print $cgi->header(), $cgi->start_html(), "\n", $cgi->end_html(), "\n"; }

PS: Package source code is on CPAN.

Replies are listed 'Best First'.
Re: Win32::MMF (Windows Memory Mapped File) Support for Perl
by NetWallah (Canon) on Feb 06, 2004 at 00:35 UTC
    This module looks promising. Thanks for providing the source - FYI - it compiled, installed and tested flawlessly on my XP machine using the dot net compiler.

    I'm sure you are probably already considering providing an object-interface for this module - just in case you are not, let me be the first to whine on this subject... or even better - if I find time, I'll try to write an OO wrapper for this module. Let me know if your or anyone else has already started on such a persuit.

    The Interface I'm thinking of would look like this:

    my $MM = Win32::MMF->new("My (optional) Unique Namespace name"); # If unspecified, a new GUID would be used for namespace # Fork the child here -- Parent & Child share $MM. # In Parent ---- my $view = $MM->CreateView("Optional Unique Name"); my $err=$view->lock([$TIMEOUT]); # Optional, Prevent others from Writi +ng $view->write($Some_Scalar_Or_Ref,[$lock,$TIMEOUT]); $view->unlock(); $view->close(); $MM->close(); #--- In Child -- my $view = $MM->GetView("Optional Unique Name", \&$Optional_CallBack_R +ef); # Line above blocks if callback is not specified my $Data = $view->read(); $view->close(); $MM->close();

    "When you are faced with a dilemma, might as well make dilemmanade. "

      Hi NetWallah. Thanks for testing the module for me. :-D

      And thanks for sharing the idea of OO wrapper too. Yes I am in the process of writing the OO wrapper for the module at the moment. I will take your suggestion into consideration for the look and feel of the OO wrapper.

      I am thinking about the semaphore issue at the moment. I am thinking about whether to use the Win32::Semaphore module to implement the lock, or to roll my own inside the module XS. I am thinking about rolling my own to reduce the dependency on other modules.

Re: Win32::MMF - Memory Mapped File Support for Perl
by xiper (Friar) on Feb 09, 2004 at 06:06 UTC
    I have been watching the recent development of this module with much interest - it fills a hole nicely on win32 that has sorely been missing (not to mention perfectly answers my recent question). I downloaded and compiled v0.03 today (after installing VS.net... whew!), but ran out of time to play with it. Now i see v0.04 is on CPAN... i'll have a look tomorrow.

    I have a couple of suggestions:

  • a tie() interface
    I can't imagine it'd be much more work, but an (additional) interface that would allow you to tie() an arbitrarily complex variable to shared memory would allow cross-platform developers to simply
    tie( $variable, $^O eq 'MSWin32' ? 'Win32::MMF' : 'IPC::Shareable', { ... } );
    Ideally it would accept the same options as IPC::Shareable, even if they're just aliases. I know this is very similar to what the module currently does, but having the tie keyword in your code makes it easier to understand what's happening. And i think this would be an even more 'programmer-friendly' interface (nothing wrong with the current one), as it hides virtually _all_ complexity. Tie a hash, fork, update it in one process, the other process can see the changes. Or tie separately in two different processes. Also makes for easier 'inter-process signalling' (via do{} while ! $share, or Tk's waitVariable()).

  • binary and/or ppm package
    Given this is win32 we're talking about, it might be a good idea to host a binary and/or a ppm package somewhere for those that don't have access to multi-billion-dollar windows compilers and/or the knowledge required. :) Just a thought.

    - ><iper

    use japh; print;
      Hi xiper, thanks for the positive feedback and excellent suggestions!

      I have added the tie() interface to my module and released version 0.05 to CPAN, plus I have fixed a bug with MMF memory allocation and recycling, and fixed a bug with variable deletion.

      The following is what you can do with the new version:
      - Everything you can do with version 0.04, plus ... use strict; use warnings; use Data::Dumper; use Win32::MMF::Shareable qw/Debug/; # Must initialize namespace before use Win32::MMF::Shareable::Init( -namespace => 'MySharedmem' ); # Tie variables to shared namespace tie my $shared, "Win32::MMF::Shareable", '$shared'; tie my @shared, "Win32::MMF::Shareable", '@shared'; tie my %shared, "Win32::MMF::Shareable", '%shared'; tie my $sh2, "Win32::MMF::Shareable", '$shared'; tie my @sh2, "Win32::MMF::Shareable", '@shared'; tie my %sh2, "Win32::MMF::Shareable", '%shared'; # as scalar $shared = "Hello world"; # as list @shared = (); for (0..3) { $shared[$_] = "$_" x ($_ + 1); } # as hash %shared = @shared; Debug(); print Dumper(\@sh2), "\n"; print Dumper(\%sh2), "\n"; print Dumper(\$sh2), "\n"; # iteration test foreach (sort keys %sh2) { print "$_ => $sh2{$_}\n"; } foreach (sort values %sh2) { print "$_\n"; } # hash slice test my @keys = keys %sh2; my @values = (0 .. $#keys); @sh2{@keys} = @values; print Dumper(\%sh2);

      :-)
        First of all, let me say this is an excellent module that is rapidly getting better. I have spent a couple of hours testing and thinking about this module and hopefully come up with some constructive comments on its current functionality. Please keep in mind these are only suggestions and very open to debate.

        General interface

        IPC::Shareable has the following basic interface:

        use IPC::Shareable; # simple method tie( $scalar, 'IPC::Shareable', 'varid' ); # complex method tie( $scalar, 'IPC::Shareable', 'varid', { ...options... } ); # OR tie( $scalar, 'IPC::Shareable', { key => 'varid', ...options... } );
        where you can do a simple 3-arg tie() and that's all, or optionally pass a hashref of options. Could i suggest a similar interface for Win32::MMF::Shareable:
        use Win32::MMF::Shareable; # simple method tie( $scalar, 'Win32::MMF::Shareable', 'varid' ); # automatically creates a namespace using default options # complex method tie( $scalar, 'Win32::MMF::Shareable', 'varid', { ...options... } ); +# OR tie( $scalar, 'Win32::MMF::Shareable', { key => 'varid', ...options... + } ); # for compatibility
        Rather than having to call Win32::MMF::Shareable::Init to set things like namespace (see following section), size, autolock, etc, use defaults for everything and pass a hashref of option overrides to tie(). To avoid having to set the same options for each tie call every time, change Init() to setdefaults() so it works something like this:
        use Win32::MMF::Shareable; # BAD tie( $bigvar1, 'Win32::MMF::Shareable', 'bigvar1', { autolock => 0, si +ze => 1024 ** 2 } ); tie( $bigvar2, 'Win32::MMF::Shareable', 'bigvar2', { autolock => 0, si +ze => 1024 ** 2 } ); tie( $bigvar3, 'Win32::MMF::Shareable', 'bigvar3', { autolock => 0, si +ze => 1024 ** 2 } ); # GOOD Win32::MMF::Shareable->setdefaults( { autolock => 0, size => 1024 ** 2 + } ); tie( $bigvar1, 'Win32::MMF::Shareable', 'bigvar1' ); tie( $bigvar2, 'Win32::MMF::Shareable', 'bigvar2' ); tie( $bigvar3, 'Win32::MMF::Shareable', 'bigvar3' );
        If you tie without any options you get the default values for each option. Optionally, you can call Win32::MMF::Shareable->setdefaults() to change the defaults for any future calls to tie. This way you have the best of simplicity and flexibility.

        Thoughts on namespaces

        IPC::Shareable uses a single identifier to reference a shared variable (refereed to as GLUE or key in the docs), where as Win32::MMF has two levels, a namespace and an identifier within the namespace. While this works well in Win32::MMF, as the namespace can be the object and you can have multiple items within the namespace, it becomes somewhat redundant when you tie a single variable with Win32::MMF::Shareable.

        tie( $scalar, 'Win32::MMF::Shareable', { namespace => 'ns1', key => 'f +oo' } ); tie( $scalar, 'Win32::MMF::Shareable', { namespace => 'ns1', key => 'b +ar' } ); tie( $scalar, 'Win32::MMF::Shareable', { namespace => 'ns2', key => 'f +oo' } ); tie( $scalar, 'Win32::MMF::Shareable', { namespace => 'ns2', key => 'b +ar' } );
        My suggestion would be to make the namespace option in Win32::MMF::Shareable optional, and have it transparently create and use the default namespace 'shareable'. This eliminates redundancy, still allows the user to override it if necessary, and allows users to mix-n-match the two modules:
        # Process 1 use Win32::MMF::Shareable; my @queue; tie( @queue, 'Win32::MMF::Shareable', 'queue' ); @queue = qw( 8265 6201 7548 2165 7892 3546 3426 6246 ); # Process 2 use Win32::MMF; my $ns = Win32::MMF->new( -namespace => "my_namespace" ); $ns->setvar( fredsname => 'fred' ); my $fredsname = $ns->getvar( 'fredsname' ); # etc... as normal # get the current value of another processes tied variable! my $server = Win32::MMF->new( -namespace => "shareable", reuse => 1 ); + my @current_queue = $server->getvar( 'queue' );
        Options

        The two modules use different flags to control when a particular 'varid' is created or mapped to an existing one. IPC::Shareable uses create & exclusive flags, while Win32::MMF has a single reuse flag, and they behave as follows (bold is the default):

        IPC::Shareable
        create flag exclusive flag varid doesn't exist varid already exists
        0 (unused) croaks maps to varid
        1 0 creates new varid maps to varid
        1 1 creates new varid croaks

        Win32::MMF
        reuse flag varid doesn't exist varid already exists
        0 creates new varid maps to varid
        1 fails silently maps to varid

        Currently you can't specify for Win32::MMF to create exclusively – failing if the varid already exists. How about something like the following, which handles all 3 cases simply:

        connect flag varid doesn't exist varid already exists
        undef (or "auto"?) creates new varid maps to varid
        1 fails silently maps to varid
        0 creates new varid fails silently

        tie( $var, 'Win32::MMF::Shareable', 'var' ); # OR tie( $var, 'Win32::MMF::Shareable', 'var', { connect => 'auto' } ); # + map if already available, otherwise create tie( $var, 'Win32::MMF::Shareable', 'var', { connect => 1 } ); # must +already exist tie( $var, 'Win32::MMF::Shareable', 'var', { connect => 0 } ); # must +not already exist
        To maintain compatibility with IPC::Shareable you would still need to accept 'create' and 'exclusive' flags (but overridden by 'connect' if present) and convert them to the appropriate connect flag.

        The other IPC::Shareable options are:

      • key – same as your 'varid', suggest you use this, or at least accept it as an alias
      • mode – redundant under win32, accept but ignore
      • destroy – removes shared data after the process exits, wether another process is using it or not. Not sure why one would use this, accept but ignore (unless you feel like implementing it)
      • size – same as your size

        Tied object

        Now that we tie our shared variable, we no longer have access to an object which means we lose functionality of lock(), unlock(), debug(), and anything else you decide to implement in the future. Other tie() modules overcome this by returning an object from the call to tie(), which would allow things like:

        $data = tie( %data, 'Win32::MMF::Shareable', 'data', { timeout => 60 } + ); if( $data->{error} ) { die( “can't tie data: ” . $data->{errstr} ) } +# maybe? or just return undef $data{min} = 0; $data{max} = 500; $data{current} = 0; $data->debug() if $debug_flag; # each process must do blocks of 10 at a time { $data->lock(); # timeout 60 seconds for( $data{end} = $data{current} + 10; $data{current} < $data{end}; +$data{current}++ ) { # do job number $data{current} } $data->unlock(); }

        Other thoughts

      • IPC::Shareable uses Storable to serialise its data – you may wish to consider switching from Data::Dumper to maintain consistancy. Not to mention Data::Dumper has some issues in what it can serialise. Maybe even provide an option for the user so they can choose.

      • The two modules Win32::MMF and Win32::MMF::Shareable, while implemented in a similar way, behave quiet differently to the user. It may be a good idea to document each separately to avoid confusion.

      • In the pod: "Because memory and variable management are managed internally by the Win32::MMF module, you do not need to specify how much memory is required by the variable." Is this correct? You don't have to specify a size for a tied variable? Will it automatically expand if the data grows? If so, why not have this for non-tied variables as well?

      • How does your module handle the issue of when to clear the data in shared memory? I assume that it's when the last 'connected' process exits or tied variable falls out of scope, correct? What about if the process is killed (via a SIGTERM)? Any possible memory leaks? (I assume you've already covered these issues, just interested to know).

        *phew* I thinks that's about it... feedback welcome!

        - ><iper

        use japh; print;
Re: (Updated Again) Win32::MMF - Memory Mapped File Support for Perl
by xiper (Friar) on Feb 16, 2004 at 01:58 UTC
    Not sure if this is specific to Win32::MMF::Shareable (v0.07), but when you $mw->waitVariable( \$tied_var ) while using Tk, it doesn't resume when $tied_var is updated in another process. Like so:
    use strict; use Tk; use Win32::MMF::Shareable; my $wait; tie( $wait, 'Win32::MMF::Shareable', 'wait' ); $wait = 0; my $pid; unless( $pid = fork ) { $SIG{HUP} = \&stop; while( 1 ) { sleep 1 } } my $mw = MainWindow->new; $mw->Button( -text => '$wait', -command => sub { pri +nt "\$wait is $wait\n" } )->pack; $mw->Button( -text => 'waitVariable( \\$wait )', -command => \&start ) +->pack; $mw->Button( -text => 'local $wait++', -command => \&stop ) +->pack; $mw->Button( -text => 'remote $wait++', -command => sub{ kill +( 'HUP', $pid ) } )->pack; MainLoop; kill( 'TERM', $pid ); sub start { print "waiting for \$wait (was $wait)\n"; $mw->waitVariable( \$wait ); print "finished waiting for \$wait (is now $wait)\n"; } sub stop { print "\$wait++ by pid $$\n"; $wait++; } __END__ # [\d] is button pressed $wait is 0 # [1] start on 0 waiting for $wait (was 0) # [2] start waiting $wait++ by pid 2372 # [3] increment locally finished waiting for $wait (is now 1) # ... all okay waiting for $wait (was 1) # [2] wait again $wait++ by pid -2184 # [4] increment remotely # ... nothing happens $wait is 2 # [1] but it has been increment +ed $wait++ by pid 2372 # [3] increment locally again finished waiting for $wait (is now 3) # ... and it works

    - ><iper

    use japh; print;

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: CUFP [id://326785]
Approved by broquaint
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (2)
As of 2024-04-20 09:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found