Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re: Looking for alternative for IPC::Shareable (or increase size)

by stevieb (Canon)
on Aug 06, 2020 at 00:55 UTC ( [id://11120385]=note: print w/replies, xml ) Need Help??


in reply to Looking for alternative for IPC::Shareable (or increase size)

Can you please show some code that reproduces what you want to achieve, but is failing?

I received permission last year to push new updates to IPC::Shareable (amongst a couple of other memory-sharing distributions). I have numerous updates in the queue, but I haven't uploaded a new version yet.

Perhaps if you could give me a couple of examples, it might intrigue and interest me enough to finish the work I was doing, and incorporate what you want at the same time.

If I knew some context and was able to see some of the critical code, I might be able to recommend an existing solution to your dilemma even.

I used, was granted author permissions on, and have modified a few modules that allow sharing of data/variables etc between processes, threads and even over the network.

Replies are listed 'Best First'.
Re^2: Looking for alternative for IPC::Shareable (or increase size)
by DomX (Novice) on Aug 06, 2020 at 07:42 UTC
    Dear stevib,

    I already discovered some strings with a length of 84_217 characters, which I try to insert into the IPC::Shareable array @sender, but I do this very often, but only on this I get lists of arrays, I can't predict of length...

    This is a snipped:
    my $allowed_childs = 10; <other code> ## Main update agent my @agency = (); # Agents (forks) my $recID = &gen_random(4); # IPC key as $transID, but o +nly visible for childs of the update agent # Separate transfer array for agents (sub-forks of fork) my $update_knot = tie(my @receiver, 'IPC::Shareable', $recI +D, { create => 1, mode => 0600, destroy => 1, size => IPC::Shareable::SHM_BUFSIZ(), #size => 131072 * 2, }); # Array to store what was transfered through @receiver, becaus +e IPC-tied arrays are very small my @storage = (); # Transfer child elements to @storage my $_sub_gatherer = sub { my $allowed_childs = shift; if ( &wait_children($allowed_childs, \@agency) ) { my $knot = tie(my @cleaner, 'IPC::Shareable', $recI +D); $knot->shlock; push(@storage, splice(@cleaner, 0, scalar(@cleaner) - +1)); $knot->shunlock; return(1); } return(0); }; <other code> # Get region dependend type ids foreach my $region ( @regions ) { my $agent = fork; # Parent if ( defined($agent) && $agent ) { push(@agency, $agent); &debug_out("get_region_types(): Starte +d download agent $agent"); #Time::HiRes::sleep(0.01); # WO +RK Maybe no longer needed because of dl_lock() #&{$_sub_gatherer}(sprintf("%.0f", $ma +x_forks * 0.5)); &{$_sub_gatherer}(1); # Report to main window $perc_count += 15 / scalar(@regions +); $ac_knot->shlock; $agent_carrier{progress_percent} = +$perc_count / &{$_sub_percent_base}(); $ac_knot->shunlock; } # Child elsif ( defined($agent) ) { srand(); my @region_market_types = &download +er(0, "/markets/$region/types/"); if ( @region_market_types ) { my $knot = tie(my @sender, 'IPC +::Shareable', $recID); $knot->shlock; push(@sender, encode_json(\@region +_market_types)); # Here the failure occures: Length of shared data ex +ceeds shared segment size at ./myprogram.pl line 1210. $knot->shunlock; } exit(0); } else { &debug_out( "update_types(): Can't fork", ); &pprop_exit(); } } # Wait for children &wait_children(1, \@agency); <other code> sub wait_children { my $left = shift; my $childs = shift; my $waiting = 0; if ( $left < 1 ) { $left = 1; } while ( scalar(@{$childs}) >= $left ) { $waiting = 1; Time::HiRes::sleep(0.1); @{$childs} = grep { kill(0 => $_) } @{$childs}; } return($waiting); }

      The man page for shmctl(2) doesn't give a way to resize an existing shared memory segment, so you can't expand it with any of the shared memory modules, since it's a limitation of the OS.

      It might be possible with mmap(2), but if other process have already mapped the memory area, then they would have to be notified of the change and unmap/map the memory again. This implies that it would have to be a file on disk so that other processes can unmap/map the same data.

      Otherwise, it's has to be something that copies the data between the processes.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11120385]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others meditating upon the Monastery: (6)
As of 2024-04-19 12:45 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found