Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I seem to be encountering some kind of overflow bug with vec. To wit:

memtime perl -e '$h = ""; $x = (2**31)-2; vec($h,$x,8) = 0;'
Exit [0]
0.94 user, 1.34 system, 2.30 elapsed -- Max VSize = 2115872KB, Max RSS
+ = 2042140KB

[download]

Runs as expected, but: 

memtime perl -e '$h = ""; $x = (2**31)-1; vec($h,$x,8) = 0;'
Out of memory!
Exit [1]
0.00 user, 0.00 system, 0.10 elapsed -- Max VSize = 3720KB, Max RSS = 
+192KB

[download]

Seems to ask for too much memory, and 

memtime perl -e '$h = ""; $x = (2**31); vec($h,$x,8) = 0;'
Negative offset to vec in lvalue context at -e line 1.
Exit [255]
0.00 user, 0.00 system, 0.10 elapsed -- Max VSize = 3720KB, Max RSS = 
+128KB

[download]

has another problem altogether.

Is this a problem with vec? My Perl build? My syntax? Any workarounds?

Replies are listed 'Best First'.
Re: vec overflow?
by BrowserUk (Patriarch) on Jun 25, 2013 at 00:58 UTC
    Is this a problem with vec? My Perl build? My syntax? Any workarounds?

    It's a problem with vec. Internally it is using a signed 32-bit values, even on 64-bit builds.

    One workaround I've used is to break the vector into subsections and use bitmasking to select the appropriate one:

    @vecs = ( chr(0)x2**29 ) x 8;;

$n = 2**31; 

vec( $vecs[ $n >> 29 ], $n & 0x1fffffff, 1 ) = 1;;

    [download]

    Not very satisfactory, but workable.

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      Thanks very much. Good to know.
[reply]

        Another workaround that is useful if your algorithm benefits from having a single contiguous bitvector -- for example if you want to count the set bits quickly using my $popcount = unpack '%32b*', $bitvector; -- is to nest calls to vec. Eg:

        vec( vec( $bitvector, $n >> 5, 64 ), $n & 0x1f, 1 ) = 1;

        [download]

        Theoretically, as neither offset breaches the 2**31-1 barrier, this can allow you to address bitvectors up to 16GB/137 billion bits, though I don't have enough memory to try it.

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
Re: vec overflow?
by LanX (Saint) on Jun 25, 2013 at 00:48 UTC
    OFFSET is a signed integer and most Perl compilations restrict them to a maximum of 31 bits.

    If you need a larger OFFSET you can switch to a 64 bit platform.¹

    vec EXPR,OFFSET,BITS
               Treats the string in EXPR as a bit vector made up of elements
               of width BITS, and returns the value of the element specified
               by OFFSET as an unsigned integer. ...

    Or just split the EXPR into smaller chunks and do basic arithmetics for calculating OFFSET.

    Cheers Rolf

    ( addicted to the Perl Programming Language)

    update

    ¹) doesn't help, according to BrowserUk

    update

    see older discussions and proposed workarounds

    update

    this workaround doesn't look efficient, since substr is an lvalue you can use constructs like 

    vec ( substr( $str, OFFSET_HIGH, 2**31), OFFSET_LOW, BITS ) = NEW

    [download]

    (well if substr allows OFFSETs that big ... can't test!)

[reply]
[d/l]
[select]
      this workaround doesn't look efficient

      It is. It takes an lvalue reference (a scalar value) and passes that to vec. It is marginally less efficient than vec( substr( ... ), ... );, but not so much that you would be able to detect it.

      However, substr can only provide an lvalue for byte-sized units; where vec can provide an lvalue for 8 byte units.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
        Ok, I had errors testing it out in the debugger, but turned out that just the automatic Data::Dump couldn't handle lvalues.

        Asa suggestion:

        I think using a foreach alias like shown in the docs for substr is a clearer alternative to lvalue-refs.

        Cheers Rolf

        ( addicted to the Perl Programming Language)

[reply]

Back to Seekers of Perl Wisdom