Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Passing a very large string by reference to a c library

by dino (Sexton)
on Jul 12, 2001 at 16:28 UTC ( #96005=perlquestion: print w/replies, xml ) Need Help??

dino has asked for the wisdom of the Perl Monks concerning the following question:


I'm building a c based binary search routine in the pursuit of speed. I need to pass to it a large string by reference and I'm not sure how to do it so that no copying occurs, using the xs interface

Suggestions Please (snippets of xs file would be very instructive)


  • Comment on Passing a very large string by reference to a c library

Replies are listed 'Best First'.
Re: Passing a very large string by reference to a c library
by jeroenes (Priest) on Jul 12, 2001 at 16:49 UTC
    Here you have a snippet:
    SV * _smooth2( yp, wp, lambda_in, m ) char * yp char * wp SV * lambda_in int m CODE: { int i, i1, i2, ip; double c[MMAX+1], d[MMAX+1], e[MMAX+1], w[MMAX+1], y[MMAX+1], z[MMAX+1], lambda; double * ya; double * wa; ya = (double *) yp; wa = (double *) wp;
    You see, the c uses pointers to address the strings. Char is the default type, you need to convert inside the code.

    From perl, I call:

    Smooth::_smooth2( $py, $pw, $lambda, scalar( @$yref ) );
    Where the $py and $pw are strings. XS automagically builds the pointers for you.

    Hope this helps,

    "We are not alone"(FZ)
    Update: Because the c-code sees only the pointers, no copy is made. To be sure, I tested it with a fresh module: TestMemXS:

    #XS #include "EXTERN.h" #include "perl.h" #include "XSUB.h" #include "unistd.h" MODULE = TestMemXS PACKAGE = TestMemXS void _test( yp ) char * yp CODE: { double * ya; ya = (double *) yp; sleep( 1000 ); } ------------------- .... sub testMem{ my $str = "\000" x 50e6; _test( $str ); } .... ------------------- perl -MTestMemXS -e 'testMem()'
    The perl process didn't exceed 55M...
      This stems from my (mistaken) belief that perl copies variables on subroutine calls. It appears that this is not true ;-). It seems that the copying may take place during the standard assignment in the subroutine body which can be avoided with a little thought. eg:
      &find_index($key, \$bigstring);
      sub find_index {
        my ($key, $rbigstring) = @_;
      &find_index($key, $bigstring);
      sub find_index {
        my ($key) = $_[0];
        my ($rbigstring) = \$_[1];

      This has the added benefit of making the sub call for both c and perl the same.

      Thanks for your help


      Mmm thanks,

      but if $pw is 50M in size then by not passing it as a ref, do the perl api routines convert it to a char * on every call?


        Perl does "pass by reference" already. No need to complicate things by taking a reference and passing that "by reference". So the above code does no copying and is really the simplest method.

                - tye (but my friends call me "Tye")
Re: Passing a very large string by reference to a c library
by Hofmator (Curate) on Jul 12, 2001 at 17:10 UTC

    Maybe example 4 of the XStutorial can help you on - it looks good but I've never really done anything in XS before...

    -- Hofmator


      This is what I've been looking at. The example doesn't use references, though example 6 does. I can try to meld both together so that I can call:

      find_index($key, \$data);

      I guess a lot depends on how SvPV works. If it copies the data then I need another method.


        I've got it to work with:

        unsigned long
        find_index (key, data)
                unsigned long         key
                SV    *               data
               STRLEN          datalen;
               char * s;
               if ((!SvROK(data))
                  || (SvTYPE(SvRV(data)) != SVt_PV))
               s = SvPV(SvRV(data), datalen);
              RETVAL = find_index (key, s, datalen);

        I'm still not sure if SvPV copies its internal data somewhere or just returns a pointer to it.

        Comments please


Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://96005]
Approved by root
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (4)
As of 2023-06-08 06:11 GMT
Find Nodes?
    Voting Booth?
    How often do you go to conferences?

    Results (29 votes). Check out past polls.