Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

IPC::Shareable sometimes leaks memory segments

by stevieb (Canon)
on Jul 08, 2019 at 20:42 UTC ( [id://11102569]=perlquestion: print w/replies, xml ) Need Help??

stevieb has asked for the wisdom of the Perl Monks concerning the following question:

Hi all,

So I've been working on a shared memory issue I'm trying to incorporate into one of my more complex distributions, and finally decided to ask for help. I've dumbed it down the best I can to replicate the problem in the minimum amount of code possible.

Code layout. A.pm contains the creation of the shared hash, I then expose the hash with our() in A.pm (and import it into B.pm). The calling script is main.pl:

main.pl <- B.pm <- A.pm

Here's a listing of how to run the code to produce the relevant outcomes:

# normal run, will finish after 7 iterations # 'ipcs -a' shows everything was cleaned up properly perl main.pl 2 2 # another normal run, but this time hit CTRL-C before the last iter # `ipcs -a` shows SIGINT handler properly cleans up perl main.pl 2 2 # hit CTRL-C # send in 1 as the first param to have B.pm die() after 5 iters # 'ipcs -a' shows everything cleaned up properly perl main.pl 1 1 # PROBLEMATIC RUN # send in three args which will die() main.pl # 'ipcs -a' shows a leak of a memory segment perl main.pl 2 2 2

Here's my cobbled together example code:

A.pm

use strict; use warnings; package A; use base 'Exporter'; our @EXPORT = qw(%hash); use IPC::Shareable; $SIG{INT} = sub { print "package A has caught the INT signal\n"; IPC::Shareable->clean_up_all; exit; }; $SIG{__DIE__} = sub { print "package A has caught the die() signal\n"; IPC::Shareable->clean_up_all; exit; }; our %hash; my $tied; BEGIN { $tied = tie %hash, 'IPC::Shareable', { key => 'test', create => 1 }; } END { IPC::Shareable->clean_up_all; } sub run { print "A: "; print "a: $hash{a}, a: $hash{b}\n"; } 1;

B.pm

use warnings; use strict; package B; use lib '.'; use A; our %hash; my $count = 0; sub run { my ($self, $x, $y) = @_; $hash{a} = $x; $hash{b} = $y; while (1){ print "B: "; print "a: $hash{a}, b: $hash{b}\n"; A->run; die "whoops!" if $count == 4 && $x == 1; last if $count == 6; $count++; sleep 1; } } 1;

main.pl

use warnings; use strict; use lib '.'; use B; print "\nneed 2 (or 3 if die()) args...\n" and exit if @ARGV < 2; print "procID: $$\n"; if (defined $ARGV[2]){ die "main script has died"; } B->run(@ARGV);

Oddly, if I run the problematic line (perl main.pl 2 2 2) more than once, only the first segment remains in ipcs -a. I would think it'd stash another entry, as it's based on procID.

In my real project, I need to catch SIGINT, die() and a myriad of other things to ensure that the shared memory segments get cleaned up appropriately in some quite complex situations, but I think if I can get help sorting out why things aren't working here, I should be able to fix the real problem.

Can anyone help me sort out why IPC::Shareable isn't cleaning up properly? If more information is needed, please don't hesitate to say so.

Note:

  • Check memory segments with ipcs -a or ipcs -ap
  • Manually clear leaked segments with ipcrm -a

Another note: If I run the problematic run (perl main.pl 1 1), then run a full good run (perl main.pl 2 2), the previously leaked segment is cleaned up. This is what is very confusing to me.

Replies are listed 'Best First'.
Re: IPC::Shareable sometimes leaks memory segments
by bliako (Monsignor) on Jul 08, 2019 at 22:47 UTC

    Perhaps you need to make explict that you want to destroy via destroy => 'yes', to the options of IPC::Shareable? This worked for me (judging by ipcs -a):

    BEGIN { $tied = tie %hash, 'IPC::Shareable', { key => 'test', create => 1, # make explicit to destroy: destroy => 'yes', }; }

    (on linux, perl 5.26.2, IPC::Shareable v 0.61 latest at this time)

    bw, bliako

      That does work correctly in this case, thanks!

      I had tried that previously in my project, but I didn't have much consistency in how I was doing things at the time (so I'm unsure if that's the reason it wasn't working). I'll apply this to the real code after I get it aligned properly, and report back.

      The issue there if I remember correctly, is I've got external processes using the same shared memory, so destroy wouldn't take until the external procs exited as well. Again, I'll try it again after I get things cleaned up. It's very likely I was holding things open by accident before.

        clean_up_all : This is a class method that provokes IPC::Shareable to remove all shared memory segments encountered by the process. Segments are removed even if they were not created by the calling process.
        
        Calling remove() on the object underlying a tie()d variable removes the associated shared memory segment. The segment is removed irrespective of whether it has the destroy option set or not and irrespective of whether the calling process created the segment.
         specifying the destroy option when tie()ing a variable coerces IPC::Shareable to remove the underlying shared memory segment when the process calling tie() exits gracefully. Note that any related shared memory segments created automagically by the use of references will also be removed.

        But this does nothing for me:

        $tied = tie %hash, 'IPC::Shareable', { key => 'test', # destroy => 'yes', create => 1 }; IPC::Shareable->clean_up_all; # but this does: # $tied->remove; exit 0;

        so I am confused.

Re: IPC::Shareable sometimes leaks memory segments
by stevieb (Canon) on Jul 15, 2019 at 19:09 UTC

    Just a bit of an update here. There were many misconceptions and misunderstandings I had about the entire shared memory situation.

    I made significant modifications to a copy of IPC::Shareable to better track resources the way I wanted to use them, and am waiting for co-auth on the distribution so I can incorporate my updates.

    However, I learned I needed other significant changes as well to go even further, so I changed the complete back end out for a new one (modified version of IPC::ShareLite, which is written in C instead of Pure Perl), and also the serializer (Storable to Serial, the latter being slightly faster, but has the ability to handle situations Storable couldn't). This means that my further changes won't be backwards-compatible, so I'll have to create a new distribution for this further work.

    I can now safely and reliably do what I set out to do (keep the data structure available persistently, ie. run one script in a window and let it exit, then start another script in another window and have it pick the data back up as if it was from disk, as well as having multiple independent scripts use the data simultaneously). We maintain registries of all segments and semaphores in use, and remove them as required.

    In order to facilitate all of this, I also had to remove some and significantly modify a lot of other code which definitely breaks backwards compatibility, so in the end, although I'll be updating IPC::Shareable, I'll have to release a new distribution as well.

    Here are some benchmarks between the original version, and the new version as it sits currently. For the purposes I need this project for, I don't need lightning speed, but I definitely need it faster than it was before (as it'll be used for tracking physical hardware changes). I still have a fair amount of work to do which'll definitely make it even faster, but so far, a 243% increase is decent already. I'm only doing 30k iterations at this time because I've got a semaphore conflict I've still got to fix, but the results are consistent over dozens of runs.

    # cmpthese (30k iterations) Rate shareable sharedhash shareable 396/s -- -71% sharedhash 1356/s 243% -- # timethese (30k iterations) Benchmark: timing 30000 iterations of shareable, shared_hash... shareable: 75 wallclock secs (41.40 usr + 32.59 sys = 73.99 CPU) @ 40 +5.46/s (n=30000) shared_hash: 22 wallclock secs (17.10 usr + 4.57 sys = 21.67 CPU) @ 1 +384.40/s (n=30000)

    Here's my simple current bench test file. As I said earlier in this thread, once the new software is done and release, I plan on writing a detailed blog post about what I've learned while going down this path.

    use warnings; use strict; use Benchmark qw(:all) ; use IPC::SharedHash; use IPC::Shareable; if (@ARGV < 1){ print "\n Need test count argument...\n\n"; exit; } my $timethis = 0; my $timethese = 0; my $cmpthese = 1; if ($timethis) { timethis($ARGV[0], \&shareable); timethis($ARGV[0], \&sharedhash); } if ($timethese) { timethese($ARGV[0], { 'shareable' => \&shareable, 'shared_hash' => \&sharedhash, }, ); } if ($cmpthese) { cmpthese($ARGV[0], { 'shareable' => \&shareable, 'sharedhash ' => \&sharedhash, }, ); } sub default { return { a => 1, b => 2, c => [qw(1 2 3)], d => {z => 26, y => 25}, }; } sub shareable { my $base_data = default(); tie my %hash, 'IPC::Shareable', 'able', { create => 1, destroy => 1 }; %hash = %$base_data; $hash{struct} = {a => [qw(b c d)]}; tied(%hash)->clean_up_all; } sub sharedhash { my $base_data = default(); tie my %hash, 'IPC::SharedHash', 'hash', { create => 1, destroy => 1 }; %hash = %$base_data; $hash{struct} = {a => [qw(b c d)]}; tied(%hash)->clean_up_all; }

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11102569]
Approved by holli
Front-paged by davies
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (4)
As of 2024-04-25 14:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found