Dear Perl Monks, I am willing to use shared memory to load in a large (4 Gb) segment of shared memory a set of ~ 4 billion packed integers. I would like to do so, because later I want several parallel workers to be able to read each nth packed integer and use it, so that from each worker, I should be able to access the 1000000th packed entry just by: 
my $packed;
my $n =1000000-1;
my $offset = $n*4; 
my $success = shmread($id, $packed, $offset, 4)

[download]
The rationale here is sharing a very large index where each packed integer refers to a specific item. ~ 3000000000 items, thanks to a simple bijective hashing function, can be mapped to integers in the range (0..4000000000) so that each worker will easily know at which offset it should look into the shared memory to retrieve the corresponding packed piece of information for each item. About ~1000000000 empty fields (because the item corresponding to that position is missing) contain just 4 nul bytes (i.e. pack("N",0)). However, when in the parent process I try just to create the shared memory object, by iteratively reading 1 million bytes from the index file and copying them into the shared memory, like this: 
use warnings;
use strict;
use IPC::SysV qw(IPC_PRIVATE IPC_RMID S_IRUSR S_IWUSR);

open(my $idx, "<", "$ARGV[0].idx") || die "cannot open data file\n $!"
+;
my $idx_size = (split(' ',`wc -c $ARGV[0].idx`))[0];

my $idx_id = shmget(IPC_PRIVATE , $idx_size, S_IRUSR | S_IWUSR) || die
+ "shmfet $!";
my $offset = 0;
foreach my $i (0..$idx_size/1000000) {
    my $n="";
    read($idx,$n,1000000);
    shmwrite($idx_id, $n, $offset, 1000000)  || die "shmwrite: $!";
    $offset +=1000000;
}

shmctl($idx_id, IPC_RMID, 0)        || die "shmctl: $!";

close $idx;

exit;

[download]
I always get an error "shmwrite: Bad address ". This happens always when writing the 2^32 th byte. So it looks like the shared memory segment perl can handle is limited to 2 Gb. however, running 
$ ipcs -m

[download]
shows that perl actually reserved a shared memory segment much larger than that (actually, the expected 4 Gb): 
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     stat
+us     
0x00000000 8454205    valerio    600        4294967296 0

[download]
I am running perl v 5.34 on an ubuntu 22.04 w/ 32 Gb RAM. perl here should be a 64-bit process: 
 
$ perl -V:archname 
archname='x86_64-linux-gnu-thread-multi';

[download]

Is this a builtin limit of perl shared memory, or is there anything that I am missing?

Thanks for your wisdom,

Valerio

In reply to Does perl have a builtin limit to the size of shared memory segments I can write to? by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.