dino has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I'm building a c based binary search routine in the pursuit of speed. I need to pass to it a large string by reference and I'm not sure how to do it so that no copying occurs, using the xs interface

Suggestions Please (snippets of xs file would be very instructive)

dino

  • Comment on Passing a very large string by reference to a c library

Replies are listed 'Best First'.
Re: Passing a very large string by reference to a c library
by jeroenes (Priest) on Jul 12, 2001 at 16:49 UTC
    Here you have a snippet: 
    SV *
_smooth2( yp, wp, lambda_in, m )
         char * yp
         char * wp
         SV * lambda_in
         int m
   CODE:
{
  int i, i1, i2, ip;
  double c[MMAX+1], d[MMAX+1], e[MMAX+1],
        w[MMAX+1], y[MMAX+1], z[MMAX+1],
        lambda;
  double * ya;
  double * wa;
  ya = (double *) yp;
  wa = (double *) wp;

    [download]
    You see, the c uses pointers to address the strings. Char is the default type, you need to convert inside the code.

    From perl, I call: 

     
Smooth::_smooth2( $py, $pw, $lambda, scalar( @$yref ) );

    [download]
    Where the $py and $pw are strings. XS automagically builds the pointers for you.

    Hope this helps,

    Jeroen
    "We are not alone"(FZ)
    Update: Because the c-code sees only the pointers, no copy is made. To be sure, I tested it with a fresh module: TestMemXS: 

    #XS
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "unistd.h"
 
MODULE = TestMemXS              PACKAGE = TestMemXS
 
 
void
_test( yp )
         char * yp
 
   CODE:
{
  double * ya;
 
  ya = (double *) yp;
 
  sleep( 1000 );
}
-------------------
#testMemXS.pm
....
sub testMem{
  my $str = "\000" x 50e6;
  _test( $str );
}
....
-------------------
perl -MTestMemXS -e 'testMem()'

    [download]
    The perl process didn't exceed 55M...
[reply]
[d/l]
[select]
      This stems from my (mistaken) belief that perl copies variables on subroutine calls. It appears that this is not true ;-). It seems that the copying may take place during the standard assignment in the subroutine body which can be avoided with a little thought. eg:
      
&find_index($key, \$bigstring);

sub find_index {
  my ($key, $rbigstring) = @_;

becomes:
&find_index($key, $bigstring);

sub find_index {
  my ($key) = $_[0];
  my ($rbigstring) = \$_[1];

      This has the added benefit of making the sub call for both c and perl the same.

      Thanks for your help

      dino

[reply]
      Mmm thanks,

      but if $pw is 50M in size then by not passing it as a ref, do the perl api routines convert it to a char * on every call?

      dino

[reply]

        Perl does "pass by reference" already. No need to complicate things by taking a reference and passing that "by reference". So the above code does no copying and is really the simplest method.

                - tye (but my friends call me "Tye")
[reply]
Re: Passing a very large string by reference to a c library
by Hofmator (Curate) on Jul 12, 2001 at 17:10 UTC

    Maybe example 4 of the XStutorial can help you on - it looks good but I've never really done anything in XS before...

    -- Hofmator

[reply]
      Thks,

      This is what I've been looking at. The example doesn't use references, though example 6 does. I can try to meld both together so that I can call:

      find_index($key, \$data);

      I guess a lot depends on how SvPV works. If it copies the data then I need another method.

      dino

[reply]
        I've got it to work with:

        unsigned long
find_index (key, data)
        unsigned long         key
        SV    *               data
    INIT:
       STRLEN          datalen;
       char * s;
       if ((!SvROK(data))
          || (SvTYPE(SvRV(data)) != SVt_PV))
       {
          XSRETURN_UNDEF;
       }
       s = SvPV(SvRV(data), datalen);
    CODE:
      RETVAL = find_index (key, s, datalen);
    OUTPUT:
      RETVAL

        I'm still not sure if SvPV copies its internal data somewhere or just returns a pointer to it.

        Comments please

        dino

[reply]

Back to Seekers of Perl Wisdom