The prototype of the respective function (doop.c, line 741) suggests the offset is in fact 32-bit:

UV Perl_do_vecget(pTHX_ SV *sv, I32 offset, I32 size)
[download]

at least if the type I32 always is 32-bit — which it seems to be...  as it turned out in a recent thread.

(As it's a signed int, the largest positive value is 2³¹-1, or 2,147,483,647.)

Perl_do_vecget(pTHX_ SV *sv, I32 offset, I32 size)
[download]

I don't know if there is any reason to be using I32 for indexes instead of IVs (native ints or bigger). Vectors are not the unique data structure with that limitation, the I32 type is used pervasively inside perl code for index values, for instance, arrays or substr also use 32 bit indexes.

The work around is to create your own XS vector module with support for 64bit indexes, but anyway, fill a bug report with perlbug, and it may be fixed for 5.12!

or sooner ( 5.10.1)

Eliminating all the I32 indexes on the interpreter would break binary compatibility, so it would be very unlikely to happen on the 5.10 branch. Though, just fixing vec could be.

Until a better version of vec is available, you might try something like this:

sub myvec(\$$$) :lvalue  {
    use constant TWO_GB => 2**31;
    my( $ref, $offset, $bits ) = @_;
    if( $offset > TWO_GB - 1 ) {
        $offset -= TWO_GB;
        $ref = \substr $$ref, ( TWO_GB * $bits ) / 8;
    }
    CORE::vec( $$ref, $offset, $bits );
}
[download]

Which should be reasonably efficient as it avoids copying the huge string. If you wanted to get fancy in anticipation of the fixed version, you could stick it in a module and export it as CORE::GLOBAL::vec.

The above is untested at the transition limit as I don't have enough memory to create strings that big. You might want to look closely at that TWO_GB - 1...

Dear Monks,
thank you for all your help! I used the same workaround (working with different chunks), albeit nowhere near as elegant as that - I will use this from now on. (I am on a 64 bit machine - I just had thought that when using a 64 bit version of perl everything would be running on 64 bits.)
thank you,
bop

I am on a 64 bit machine

If there is any chance that your strings will get bigger than 4GB, I think you just need to change the if for a while and the rest would take care of itself:

sub myvec(\$$$) :lvalue  {
    use constant TWO_GB => 2**31;
    my( $ref, $offset, $bits ) = @_;
    while( $offset > TWO_GB - 1 ) {
        $offset -= TWO_GB;
        $ref = \substr $$ref, ( TWO_GB * $bits ) / 8;
    }
    CORE::vec( $$ref, $offset, $bits );
}
[download]

but again that's untested, so check it and convince yourself.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

-sam

Did you? It appears to me that Bit::Vector's new() takes a number of bits via an argument of type N_int which is defined as "unsigned int" which is likely 32 bits which means 2**32 bits or 2**32/8 bytes or 512MB, not even 2GB.

I went to try Bit::Vector but ended up killing it before it finished allocating the (it appears) 512MB of memory.

- tye        

Too bad it didn't work. I think at the very least you'll need to be on a 64-bit platform where an int is 64-bits. That might let even vec() work since Perl's "32 bit" type might end up being 64 bits anyway.

I suppose the obvious fix here is to split your data into chunks and make access a two step process - pick the right chunk and then vec() into it.

-sam