in reply to Re: Win32::MMF - Memory Mapped File Support for Perl
in thread (Updated Again) Win32::MMF - Memory Mapped File Support for Perl

Hi xiper, thanks for the positive feedback and excellent suggestions!

I have added the tie() interface to my module and released version 0.05 to CPAN, plus I have fixed a bug with MMF memory allocation and recycling, and fixed a bug with variable deletion.

The following is what you can do with the new version:
- Everything you can do with version 0.04, plus ...

use strict;
use warnings;
use Data::Dumper;
use Win32::MMF::Shareable qw/Debug/;

# Must initialize namespace before use
Win32::MMF::Shareable::Init( -namespace => 'MySharedmem' );

# Tie variables to shared namespace
tie my $shared, "Win32::MMF::Shareable", '$shared';
tie my @shared, "Win32::MMF::Shareable", '@shared';
tie my %shared, "Win32::MMF::Shareable", '%shared';

tie my $sh2, "Win32::MMF::Shareable", '$shared';
tie my @sh2, "Win32::MMF::Shareable", '@shared';
tie my %sh2, "Win32::MMF::Shareable", '%shared';

# as scalar
$shared = "Hello world";

# as list
@shared = ();
for (0..3) {
    $shared[$_] = "$_" x ($_ + 1);
}

# as hash
%shared = @shared;

Debug();

print Dumper(\@sh2), "\n";
print Dumper(\%sh2), "\n";
print Dumper(\$sh2), "\n";

# iteration test
foreach (sort keys %sh2) {
    print "$_ => $sh2{$_}\n";
}

foreach (sort values %sh2) {
    print "$_\n";
}

# hash slice test
my @keys = keys %sh2;
my @values = (0 .. $#keys);
@sh2{@keys} = @values;

print Dumper(\%sh2);

[download]

:-)

Replies are listed 'Best First'.
Re: Re: Re: Win32::MMF - Memory Mapped File Support for Perl
by xiper (Friar) on Feb 11, 2004 at 04:31 UTC
    First of all, let me say this is an excellent module that is rapidly getting better. I have spent a couple of hours testing and thinking about this module and hopefully come up with some constructive comments on its current functionality. Please keep in mind these are only suggestions and very open to debate.

    General interface

    IPC::Shareable has the following basic interface: 

    use IPC::Shareable;

# simple method
tie( $scalar, 'IPC::Shareable', 'varid' );

# complex method
tie( $scalar, 'IPC::Shareable', 'varid', { ...options... } );  # OR
tie( $scalar, 'IPC::Shareable', { key => 'varid', ...options... } );

    [download]
    where you can do a simple 3-arg tie() and that's all, or optionally pass a hashref of options. Could i suggest a similar interface for Win32::MMF::Shareable: 
    use Win32::MMF::Shareable;

# simple method
tie( $scalar, 'Win32::MMF::Shareable', 'varid' );
# automatically creates a namespace using default options

# complex method
tie( $scalar, 'Win32::MMF::Shareable', 'varid', { ...options... } );  
+# OR
tie( $scalar, 'Win32::MMF::Shareable', { key => 'varid', ...options...
+ } );
# for compatibility

    [download]
    Rather than having to call Win32::MMF::Shareable::Init to set things like namespace (see following section), size, autolock, etc, use defaults for everything and pass a hashref of option overrides to tie(). To avoid having to set the same options for each tie call every time, change Init() to setdefaults() so it works something like this: 
    use Win32::MMF::Shareable;

# BAD

tie( $bigvar1, 'Win32::MMF::Shareable', 'bigvar1', { autolock => 0, si
+ze => 1024 ** 2 } );
tie( $bigvar2, 'Win32::MMF::Shareable', 'bigvar2', { autolock => 0, si
+ze => 1024 ** 2 } );
tie( $bigvar3, 'Win32::MMF::Shareable', 'bigvar3', { autolock => 0, si
+ze => 1024 ** 2 } );

# GOOD

Win32::MMF::Shareable->setdefaults( { autolock => 0, size => 1024 ** 2
+ } );

tie( $bigvar1, 'Win32::MMF::Shareable', 'bigvar1' );
tie( $bigvar2, 'Win32::MMF::Shareable', 'bigvar2' );
tie( $bigvar3, 'Win32::MMF::Shareable', 'bigvar3' );

    [download]
    If you tie without any options you get the default values for each option. Optionally, you can call Win32::MMF::Shareable->setdefaults() to change the defaults for any future calls to tie. This way you have the best of simplicity and flexibility.

    Thoughts on namespaces

    IPC::Shareable uses a single identifier to reference a shared variable (refereed to as GLUE or key in the docs), where as Win32::MMF has two levels, a namespace and an identifier within the namespace. While this works well in Win32::MMF, as the namespace can be the object and you can have multiple items within the namespace, it becomes somewhat redundant when you tie a single variable with Win32::MMF::Shareable. 

    tie( $scalar, 'Win32::MMF::Shareable', { namespace => 'ns1', key => 'f
+oo' } );
tie( $scalar, 'Win32::MMF::Shareable', { namespace => 'ns1', key => 'b
+ar' } );
tie( $scalar, 'Win32::MMF::Shareable', { namespace => 'ns2', key => 'f
+oo' } );
tie( $scalar, 'Win32::MMF::Shareable', { namespace => 'ns2', key => 'b
+ar' } );

    [download]
    My suggestion would be to make the namespace option in Win32::MMF::Shareable optional, and have it transparently create and use the default namespace 'shareable'. This eliminates redundancy, still allows the user to override it if necessary, and allows users to mix-n-match the two modules: 
    # Process 1
use Win32::MMF::Shareable;

my @queue;
tie( @queue, 'Win32::MMF::Shareable', 'queue' );
@queue = qw( 8265 6201 7548 2165 7892 3546 3426 6246 );


# Process 2
use Win32::MMF;

my $ns = Win32::MMF->new( -namespace => "my_namespace" );
$ns->setvar( fredsname => 'fred' );
my $fredsname = $ns->getvar( 'fredsname' );
# etc... as normal

# get the current value of another processes tied variable!
my $server = Win32::MMF->new( -namespace => "shareable", reuse => 1 );
+ 
my @current_queue = $server->getvar( 'queue' );

    [download]
    Options

    The two modules use different flags to control when a particular 'varid' is created or mapped to an existing one. IPC::Shareable uses create & exclusive flags, while Win32::MMF has a single reuse flag, and they behave as follows (bold is the default):

    IPC::Shareable
    create flag exclusive flag varid doesn't exist varid already exists
    0 (unused) croaks maps to varid
    1 0 creates new varid maps to varid
    1 1 creates new varid croaks

    Win32::MMF
    reuse flag varid doesn't exist varid already exists
    0 creates new varid maps to varid
    1 fails silently maps to varid

    Currently you can't specify for Win32::MMF to create exclusively – failing if the varid already exists. How about something like the following, which handles all 3 cases simply:

    connect flag varid doesn't exist varid already exists
    undef (or "auto"?) creates new varid maps to varid
    1 fails silently maps to varid
    0 creates new varid fails silently 

    tie( $var, 'Win32::MMF::Shareable', 'var' );  # OR
tie( $var, 'Win32::MMF::Shareable', 'var', { connect => 'auto' } );  #
+ map if already available, otherwise create
tie( $var, 'Win32::MMF::Shareable', 'var', { connect => 1 } ); # must 
+already exist
tie( $var, 'Win32::MMF::Shareable', 'var', { connect => 0 } ); # must 
+not already exist

    [download]
    To maintain compatibility with IPC::Shareable you would still need to accept 'create' and 'exclusive' flags (but overridden by 'connect' if present) and convert them to the appropriate connect flag.

    The other IPC::Shareable options are:

  • key – same as your 'varid', suggest you use this, or at least accept it as an alias
  • mode – redundant under win32, accept but ignore
  • destroy – removes shared data after the process exits, wether another process is using it or not. Not sure why one would use this, accept but ignore (unless you feel like implementing it)
  • size – same as your size

    Tied object

    Now that we tie our shared variable, we no longer have access to an object which means we lose functionality of lock(), unlock(), debug(), and anything else you decide to implement in the future. Other tie() modules overcome this by returning an object from the call to tie(), which would allow things like: 

    $data = tie( %data, 'Win32::MMF::Shareable', 'data', { timeout => 60 }
+ );
if( $data->{error} ) { die( “can't tie data: ” . $data->{errstr} ) }  
+# maybe? or just return undef

$data{min} = 0;
$data{max} = 500;
$data{current} = 0;

$data->debug() if $debug_flag;

# each process must do blocks of 10 at a time
{
  $data->lock();  # timeout 60 seconds

  for( $data{end} = $data{current} + 10; $data{current} < $data{end}; 
+$data{current}++ )
  {
    # do job number $data{current}
  }

  $data->unlock();
}

    [download]

    Other thoughts

  • IPC::Shareable uses Storable to serialise its data – you may wish to consider switching from Data::Dumper to maintain consistancy. Not to mention Data::Dumper has some issues in what it can serialise. Maybe even provide an option for the user so they can choose.

  • The two modules Win32::MMF and Win32::MMF::Shareable, while implemented in a similar way, behave quiet differently to the user. It may be a good idea to document each separately to avoid confusion.

  • In the pod: "Because memory and variable management are managed internally by the Win32::MMF module, you do not need to specify how much memory is required by the variable." Is this correct? You don't have to specify a size for a tied variable? Will it automatically expand if the data grows? If so, why not have this for non-tied variables as well?

  • How does your module handle the issue of when to clear the data in shared memory? I assume that it's when the last 'connected' process exits or tied variable falls out of scope, correct? What about if the process is killed (via a SIGTERM)? Any possible memory leaks? (I assume you've already covered these issues, just interested to know).

    *phew* I thinks that's about it... feedback welcome!

    - ><iper 

    use japh; print;

    [download]
[reply]
[d/l]
[select]
      Thanks xiper++ once again for the excellent feedback! :-D

      I have made several improvements after reading your comments...

      1) no explicit Init statement required
      The default MMF namespace (shared memory) will be created if the user has not explicitly provided namespace parameters. Once initialized, all subsequent tie() will use the default namespace, unless otherwise specified in the parameter list to tie().
      use strict;
use Win32::MMF::Shareable;
use Data::Dumper;

my $ns = tie my $ref, 'Win32::MMF::Shareable', '$ref';
tie my $alias, 'Win32::MMF::Shareable', '$ref';

$ref = { a => 1, b => 2, c => 3 };
$alias->{d} = 4;
print Dumper($ref);

$ref = [ qw( a b c d e f g h ) ];
print Dumper($alias);

# $ns->debug();

      [download]
      Read more... (604 Bytes)

      2) Namespace consideration
      Win32::MMF has a slightly different design to IPC::Shareable in shared memory management. The following diagram is a view of a MMF namespace:
      +-----------------+
| Var1 definition |
+-----------------+
| Var2 definition |
+-----------------+<- Heap Top
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|-----------------|<- Watermark
|                 |
|                 |
|                 |
|                 |
+-----------------+<- Heap Bottom
| MMF Descriptor  |
+-----------------+

      [download]
      Every MMF created carries a MMF descriptor table that contains information about the size of the MMF, number of variables declared inside the namespace and heap information. Variable definition starts from the top of MMF and grows downward. The variable definition carries a pointer if there is associated data for the variable. Memory is allocated from the heap bottom and grows upward to hold the actual data. When a variable is deleted, the MMF manager deletes the variable definition and frees the memory allocated to the variable data.

      A MMF contains everything a process need to know about, because all the memory management information is carried inside the MMF itself. So that when one process adds variable, another process immediately sees it, and can use it without any additional initialization steps.

      The tie() wrapper does not directly control how a variable is stored inside the MMF. It doesn't care how much memory is required to store a particular tied variable, the MMF manager automatically shrinks or grows the memory allocated to the variable, as long as there is still enough memory left in the heap.

      # key ¨C same as your 'varid', suggest you use this, or at least accept it as an alias

      The Win32::MMF::Shareable module is designed as a drop-in replacement to IPC::Shareable under Windows, you can use it the same way as IPC::Shareable. Options like mode, destroy, are ignored. I have created the copied the following demo from then IPC::Shareable demo in the POD, replaced use IPC::Shareable to automatically switch between Win32::MMF::Shareable and IPC::Shareable on different platform, leaving the rest of the code untouched.
      use strict;
# ----------------------------------------------------------------
# code to switch between Win32::MMF::Shareable and IPC::Shareable
# automatically on different platforms
# ----------------------------------------------------------------
use vars qw/ $Shareable /;
BEGIN {
    $Shareable = ($^O eq 'MSWin32') ? 'Win32::MMF::Shareable'
                                    : 'IPC::Shareable';
    eval "require $Shareable";
    $Shareable->import();
}
# ----------------------------------------------------------------

my $glue = 'data';
my %options = (
    create    => 0,
    exclusive => 0,
    mode      => 0644,
    destroy   => 0,
    );
my %colours;
tie %colours, $Shareable, $glue, { %options } or
    die "client: tie failed\n";
foreach my $c (keys %colours) {
    print "client: these are $c: ",
        join(', ', @{$colours{$c}}), "\n";
}
delete $colours{'red'};
exit;

      [download]

      # The two modules Win32::MMF and Win32::MMF::Shareable, while implemented in a similar way, behave quiet differently to the user. It may be a good idea to document each separately to avoid confusion.

      Will do.

      # In the pod: "Because memory and variable management are managed internally by the Win32::MMF module, you do not need to specify how much memory is required by the variable." Is this correct? You don't have to specify a size for a tied variable? Will it automatically expand if the data grows? If so, why not have this for non-tied variables as well?

      That's correct, the MMF manager decides the correct amount of memory required inside the MMF to hold the object, so it can be correctly recreated from another process. This applies to untied variables too.

      # How does your module handle the issue of when to clear the data in shared memory? I assume that it's when the last 'connected' process exits or tied variable falls out of scope, correct? What about if the process is killed (via a SIGTERM)? Any possible memory leaks? (I assume you've already covered these issues, just interested to know).

      Currently it's up to Windows own memory management system to handle a killed process. Windows will destroy the MMF if all processes accessing it are killed. (A least I hope so.)

[reply]
[d/l]
[select]

In Section Cool Uses for Perl