in reply to XS: SvPVLV examples?

What I'm seeking is the lowest cost way to provide direct access to memory (mapped to a file) from Perl code.

Right off the bat, that eliminates magic. Let's compare 

substr($s, $pos, $len) = $x;

[download]
to 
substr($s, $pos, $len, $x);

[download]

This is what occurs in the assignment version:

  1. A var is created, upgraded and made magical.
  2. The value of $x is copied into the PV of the magical var
  3. The magic handler of the magical var is called.
  4. It copies the the value of the PV into $s (resizing if necessary).
  5. The magical var is freed.

In contrast, the following occurs for the 4-arg version:

  1. The value of $x is copied into $s (resizing if necessary).

None of the extra steps in the first version are particularly cheap either.

But let's say you're ok with the overhead.

it falls foul of the substr limitations you helped identify earlier in the week which means that you cannot use offsets ≥2GB.

Then substr magic is definitely not acceptable, since suffers from the same problem. 

int
Perl_magic_getsubstr(pTHX_ SV *sv, MAGIC *mg)
{
    STRLEN len;
    SV * const lsv = LvTARG(sv);
    const char * const tmps = SvPV_const(lsv,len);
    I32 offs = LvTARGOFF(sv);
    I32 rem = LvTARGLEN(sv);
    ...
}

int
Perl_magic_setsubstr(pTHX_ SV *sv, MAGIC *mg)
{
    dVAR;
    STRLEN len;
    const char * const tmps = SvPV_const(sv, len);
    SV * const lsv = LvTARG(sv);
    I32 lvoff = LvTARGOFF(sv);
    I32 lvlen = LvTARGLEN(sv);
    ...
}

[download]

Actually, the more I think about this, I think you may have been half right back there when you talked about lvalue subs

I looked in perlxs quickly, and I didn't see anything about this. Maybe adding one of the following to the associated .pm will work: 

sub xs_func1 :lvalue;
sub xs_func2 :lvalue;

[download]

or 

use attributes ();
for (qw( xs_func1 xs_func2 )) {
    no strict 'refs';
    attributes->import(__PACKAGE__, \&{$_}, 'lvalue');
}

[download]

And in any case, incrementing bytes isn't particularly useful

Sounds like you want your function to return numbers. 

my $mmap = $class->new( ..., endianess => ENDIAN_NATIVE );

++( $mmap->uint8(0x1234) );

[download]

Replies are listed 'Best First'.
Re^2: XS: SvPVLV examples?
by ikegami (Patriarch) on Sep 25, 2009 at 18:01 UTC
    And here's a sample getter and setter: 
    typedef enum {
    ENDIAN_NATIVE,
    ENDIAN_NETWORK,
    ENDIAN_LITTLE,

    ENDIAN_BIG = ENDIAN_NETWORK
} ENDIANNESS;


int
magic_get_uint16(pTHX_ SV *sv, MAGIC *mg)
{
    STRLEN len;
    SV* const src_sv = LvTARG(sv);
    const char* const src_mem = SvPV_const(src_sv, len);
    STRLEN ofs = LvTARGOFF(sv);
    ENDIANNESS endianness = (ENDIANNESS)LvTARGLEN(sv);
    U16 dst;

    /* We use a PVLV instead of storing an SV in the magic */
    PERL_UNUSED_ARG(mg);

    if (ofs >= len/2)
        croak("Index outside of mmap");

    ofs *= 2;

    if (endianness == ENDIAN_LITTLE)
        dst = (src_mem[ofs+1] << 8) | src_mem[ofs+0];
    else if (sizeof(U16) != 2 || endianness == ENDIAN_NETWORK)
        dst = (src_mem[ofs+0] << 8) | src_mem[ofs+1];
    else {
        *(((char*)&dst)+0) = src_mem[ofs+0];
        *(((char*)&dst)+1) = src_mem[ofs+1];
    }

    sv_setuv(sv, dst);

    return 0;
}


int
magic_set_uint16(pTHX_ SV *sv, MAGIC *mg)
{
    STRLEN len;
    SV* const dst_sv = LvTARG(sv);
    char* const dst_mem = SvPV(dst_sv, len);
    STRLEN ofs = LvTARGOFF(sv);
    ENDIANNESS endianness = (ENDIANNESS)LvTARGLEN(sv);
    UV num
    U16 src;

    /* We use a PVLV instead of storing an SV in the magic */
    PERL_UNUSED_ARG(mg);

    SvUV(sv);
    if (!SvIOK(sv))
        croak("Invalid value: Not a integer");

    num = SvUVX(sv);
    if (num > 65535)
        croak("Invalid value: Not a 16-bit integer");

    src = (U16)num;

    if (ofs >= len/2)
        croak("Index outside of mmap");

    ofs *= 2;

    if (endianness == ENDIAN_LITTLE) {
        dst_mem[ofs+0] = src & 0xFF;
        dst_mem[ofs+1] = src >> 8;
    }
    else if (sizeof(U16) != 2 || endianness == ENDIAN_NETWORK) {
        dst_mem[ofs+0] = src >> 8;
        dst_mem[ofs+1] = src & 0xFF;
    }
    else {
        dst_mem[ofs+0] = *(((const char*)&src)+0);
        dst_mem[ofs+1] = *(((const char*)&src)+1);
    }

    return 0;
}

    [download]

    It would simplify things to case src_mem/dst_mem to *U16, but U16 can be more than 16 bits and it introduces alignment issues.

    I made the native byte ordering big endian on systems where there is no 16-bit unsigned int.

[reply]
[d/l]
[select]
      const char* const src_mem = SvPV_const(lsv, len);

      [download]
      1. Where is SvPV_const() documented?
      2. And where does lsv come from?
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]
        1. Doesn't appear to be. It surely isn't necessary in the getter to do so much consting in the getter, and it's wrong to use SvPV_const in the setter. Fixed setter.
        2. Should be src_sv and dst_sv. Fixed.
[reply]
[d/l]
Re^2: XS: SvPVLV examples?
by BrowserUk (Patriarch) on Sep 25, 2009 at 19:29 UTC
    Right off the bat, that eliminates magic.

    I disagree. If you are dealing with numeric data in its binary form, you will always have to do some pre and post processing of the binary values if you are to combine them with normal math opcodes. The pack and unpack in that unwieldy line I posted.

    By using magic, the lvalue can appear as a normal perl variable with the appropriate numeric attributes without having to process pack/unpack templates and all the complexities they involve, or the unwieldyness of non-lvalue expressions. But I see you reached the same conclusion later.

    Then substr magic is definitely not acceptable,

    I knew that I would be using corrected versions of the substr magic set/get. To both correct the deficiencies, and simplify the logic.

    I looked in perlxs quickly, and I didn't see anything about this.

    The XS docs came up short in my searches for how to implement lvalue XS subs, but as I showed in my modifyied version of your sub above, just the addition of a lvalue prototype for the XS subs before use does work.

    I found a reference to it from back in the 5.7.x days http://www.mail-archive.com/perl5-porters@perl.org/msg89367.html. I also found a reference to an XS keyword ATTRS, but this doesn't show up in the docs anywhere.

    Sounds like you want your function to return numbers

    The idea is that the subs will return a PVLV that will act like the appropriate form of IV/UV/NV etc. but without the need to create a full-blown SV for every offset. Preferably, the endianness of a particular value will be retained within the lvalue itself so that it doesn't need to be reasserted for every use.

    BTW: Thanks for the to magic subs you posted elsewhere. One thought. It would be easier to follow if the posts were in a single contiguous thread rather than scattered around.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

      I disagree. If you are dealing with numeric data in its binary form, you will always have to do some pre and post processing of the binary values

      That has nothing to do with whether you use 

      ++( $mmap->uint8(0x1234) );

      [download]
      or the faster 
      $mmap->preinc_uint8(0x1234);

      [download]

      Now, using magic allows you to reuse existing code (like the ++ operator), so I thought you'd be willing to absorb the cost. That's why I proceeded.

      It would be easier to follow if the posts were in a single contiguous thread rather than scattered around.

      Then depth of the thread was causing problems. I took the fresh start from actually breaking out of the XY problem as a good point to restart. Your post actually forks that new branch.

[reply]
[d/l]
[select]

        How about the even faster:++ u8( $mmap, 0x12346789abcd );

        or faster and simpler 

        my $uchar = u8( $mmap, 0x12346789abcd ); 

++$$uchar if cond;

$$uchar *= 3;

        [download]

        I've seen much deeper threads and never seen it as a problem.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom