Wrapping a C shared library with Perl and XS

So, I've been asked by a couple of people now if I would take some of the experience I've gained over the last half year or so, and put together some form of tutorial on wrapping a C library, and more generally, XS. This is the first cut of that tutorial.

Relatively, I am still very new to all of this, as it's a pretty complex world. Before I started, I didn't have any real C experience, so I've been dealing with that learning curve at the same time, so I know there are better and more efficient ways of doing what I do, and would appreciate any feedback.

I'll get right to it. Here's an overview:

find something to wrap. In this case, I've written a shared C library called xswrap (I'll detail that whole procedure in the first reply to this node)
create a shell distribution that'll allow us to load our eventual XS code, which in turn has wrapped the C library
update relevant files to make things hang together
run into a function that can't be returned to Perl as-is, so we learn how to write our own C/XS wrapper so we can get what we need
package it all together into a distribution

The actual C code is irrelevant at this point, but knowing the definitions in use is, so here they are for the xswrap library:

int mult (int x, int y);
void speak (const char* str);
unsigned char* arr (); // returns (0, 1, 2)
[download]

Create a new shell distribution

I use Module::Starter:

module-starter \
    --module=XS::Wrap \
    --author="Steve Bertrand" \
    --email=steveb@cpan.org \
    --license=perl
[download]

Now change into the new XS-Wrap directory, which is the root directory of the new dist. The Perl module file is located at lib/XS/Wrap.pm. I've removed a bunch of stuff for brevity, but the shell looks something like this:

package XS::Wrap;

use warnings;
use strict;

our $VERSION = '0.01';
[download]

Create the base XS file

I use Inline::C to do this for me, as like most Perl hackers, I'm often lazy. Note the flags in use. clean_after_build tells Inline to not clean up the build directory (_Inline after build). This allows us to fetch our new .xs file. name is the name of the module we're creating this XS file for.

use warnings;
use strict;

use Inline Config =>
           disable => clean_after_build =>
           name => 'XS::Wrap';
use Inline 'C';

__END__
__C__

#include <stdio.h>
#include <xswrap.h>
[download]

The resulting XS file is located in _Inline/build/XS/Wrap/Wrap.xs. Copy it to the root directory of the dist:

cp _Inline/build/XS/Wrap/Wrap.xs .

Here's what our base XS file looks like. It doesn't do anything yet, but we'll get there:

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "INLINE.h"

#include <stdio.h>
#include <xswrap.h>

MODULE = XS::Wrap  PACKAGE = main

PROTOTYPES: DISABLE
[download]

See the PACKAGE = main there? Change main to the name of our dist, XS::Wrap.

Adding the functions from the shared library to XS

Now we need to define our C functions within the XS file. After I've done that using the C definitions for the functions above, my XS file now looks like this

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "INLINE.h"

#include <stdio.h>
#include <xswrap.h>

MODULE = XS::Wrap  PACKAGE = XS::Wrap

PROTOTYPES: DISABLE

int mult (x, y)
    int x
    int y

void speak (str)
    const char* str

unsigned char* arr ()
[download]

Note that at this point, because we're not using Inline anymore, you can remove the include for the INLINE.h header file. However, in our case, we're going to be using some Inline functionality a bit later on, so instead of removing that, copy the INLINE.h file to the same directory we copied our XS file into: cp _Inline/build/XS/Wrap/INLINE.h .

Readying the module file for use

Now we have some work to do to pull in the XS, wrap the functions, and export them. Note that you do not *need* to wrap the functions with Perl, you can export them directly as depicted in the XS file if you wish, as long as you know you don't need to add any further validation or functionality before the XS imported C function is called. I'll wrap all three. The functions that each wrapped function calls is the literal C function, as advertised through the XS file we hacked above.

use warnings;
use strict;

our $VERSION = '0.01';

require XSLoader;
XSLoader::load('XS::Wrap', $VERSION);

use Exporter qw(import);

our @EXPORT_OK = qw(
    my_mult
    my_speak
    my_arr
);

our %EXPORT_TAGS;
$EXPORT_TAGS{all} = [@EXPORT_OK];

sub my_mult {
    my ($x, $y) = @_;
    return mult($x, $y);
}
sub my_speak {
    my ($str) = @_;
    speak($str);
}
sub my_arr {
    my @array = arr();
    return @array;
}
[download]

Telling the Makefile to load the external C library

Because we're using an external shared library, we need to add a directive to the Makefile.PL file. Put the following line anywhere in the Makefile.PL's WriteMakefile() routine:

LIBS => ['-lxswrap'],
[download]

Building, installing and initial test

Let's build, install and write a test script for our new distribution.

perl Makefile.PL
make
make install
[download]

At this point, if everything works as expected, you're pretty well done. However, in the case here, we're going to unexpectedly run into some issues, and we'll need to do other things before we finalize our distribution.

Test script (example.pl). Very basic, it just tests all three wrapped functions:

use warnings;
use strict;
use feature 'say';

use XS::Wrap qw(:all);

say my_mult(5, 5);

my_speak("hello, world!\n");

my @arr = my_arr();

say $_ for @arr;
[download]

Output:

25
hello, world!
[download]

Hmmm, something is not right. The arr() C function was supposed to return an array of three elements, 0, 1, 2, but we get no output.

This is because arr() returns an unsigned char* which we can't handle correctly/directly in Perl.

In this case, I will just wrap the arr() function with a new C function (I've called it simply _arr()) that returns a real Perl array based on the output from the original C arr() function. Technically, I won't be returning anything, I'm going to just use functionality from Inline to push the list onto the stack (one element at a time), where Perl will automatically pluck it back off of the stack.

To do this, I'll be leveraging Inline again, but with a couple of changes. We change the name, and add also bring in our shared library because we need it directly now.

Returning a Perl array from a C function

use warnings;
use strict;

use Inline config =>
           disable => clean_after_build =>
           name => 'Test';
use Inline ('C' => 'DATA', libs => '-lxswrap');

print "$_\n" for _arr();

 __END__
 __C__

#include <stdio.h>
#include <xswrap.h>

void _arr (){
    unsigned char* c_array = arr();

    inline_stack_vars;
    inline_stack_reset;

    int i;

    for (i=0; i<3; i++){
        inline_stack_push(sv_2mortal(newSViv(c_array[i])));
    }

    free(c_array);
    
    inline_stack_done;
}
[download]

After I execute that Perl script, I'm left with a new XS file within the _Inline/build/Test/Test.xs.. It looks like this:

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "INLINE.h"

#include <stdio.h>
#include <xswrap.h>

void _arr (){
    unsigned char* c_array = arr();

    inline_stack_vars;
    inline_stack_reset;

    int i;

    for (i=0; i<3; i++){
        inline_stack_push(sv_2mortal(newSViv(c_array[i])));
    }

    free(c_array);

    inline_stack_done;
}


MODULE = Test  PACKAGE = main  

PROTOTYPES: DISABLE


void
_arr ()
        PREINIT:
        I32* temp;
        PPCODE:
        temp = PL_markstack_ptr++;
        _arr();
        if (PL_markstack_ptr != temp) {
          /* truly void, because dXSARGS not invoked */
          PL_markstack_ptr = temp;
          XSRETURN_EMPTY; /* return empty stack */
        }
        /* must have used dXSARGS; list context implied */
        return; /* assume stack size is correct */
[download]

We only need a couple of pieces of it, so get out your CTRL-V and CTRL-C. Here are the sections (cleaned up a bit for brevity) that we need to copy into our real Wrap.xs file.

The C portion:

void _arr (){
    unsigned char* c_array = arr();

    inline_stack_vars;
    inline_stack_reset;

    int i;

    for (i=0; i<3; i++){
        inline_stack_push(sv_2mortal(newSViv(c_array[i])));
    }
    
    free(c_array);
    
    inline_stack_done;
}
[download]

The XS portion:

void
_arr ()
        PREINIT:
        I32* temp;
        PPCODE:
        temp = PL_markstack_ptr++;
        _arr();
        if (PL_markstack_ptr != temp) {
          PL_markstack_ptr = temp;
          XSRETURN_EMPTY;
        }
        return;
[download]

The C part goes near the top of the XS file, and the XS part goes in the XS section at the bottom. Here's our full XS file after I've merged in these changes.

Finalized XS file

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "INLINE.h"

#include <stdio.h>
#include <xswrap.h>

void _arr (){
    unsigned char* c_array = arr();

    inline_stack_vars;
    inline_stack_reset;

    int i;

    for (i=0; i<3; i++){
        inline_stack_push(sv_2mortal(newSViv(c_array[i])));
    }

    free(c_array);
    
    inline_stack_done;
}

MODULE = XS::Wrap  PACKAGE = XS::Wrap

PROTOTYPES: DISABLE

int mult (x, y)
    int x
    int y

void speak (str)
    const char* str

unsigned char* arr ()

void
_arr ()
        PREINIT:
        I32* temp;
        PPCODE:
        temp = PL_markstack_ptr++;
        _arr();
        if (PL_markstack_ptr != temp) {
          PL_markstack_ptr = temp;
          XSRETURN_EMPTY;
        }
        return;
[download]

So, in our XS, we have four functions. Three that are imported directly from the C shared lib (mult(), speak() and arr()) and one new one written in C locally that wraps an imported XS function (_arr()).

We need to do a quick update to the wrapper in the module file. Change the call to arr() to _arr() in the .pm file within the my_arr() function:

sub my_arr {
    my @array = _arr();
    return @array;
}
[download]

Repeat the build/install steps, then test again:

perl example.pl
25
hello, world!

0
1
2
[download]

Cool! Our custom C wrapper for arr() works exactly how we want it to.

We're ready for release!

Creating a release of our distribution

It's very trivial to do:

rm -rf _Inline
perl Makefile.PL
make
make test
make manifest
make install
make dist
[download]

Of course, you have written all of your POD and unit tests before reaching this point, but I digress :)

I've also posted this at blogs.perl.org.

update: I want to thank all of the Monks here who have provided me help, feedback, advice and in a couple of cases, some ego-kicking. I will not name said Monks because I'm very afraid of leaving someone out, but you know who you are.

Comment on Wrapping a C shared library with Perl and XS Select or Download Code

Replies are listed 'Best First'.
Re: Wrapping a C shared library with Perl and XS by syphilis (Archbishop) on Mar 18, 2017 at 03:01 UTC
I can't immediately see how: `MODULE = Test PACKAGE = main` transforms into `MODULE = XS::Wrap PACKAGE = XS::Wrap`. Is that done by hand ? I'd attack the problem using InlineX::C2XS, but few others seem to share my enthusiasm for that (with good reasons, no doubt). That is, for me, one way is to cd to some directory and create ./src/Wrap.c, where that file contains your C code: `#include <stdio.h> #include <xswrap.h> void _arr (){ unsigned char* c_array = arr(); inline_stack_vars; inline_stack_reset; int i; for (i=0; i<3; i++){ inline_stack_push(sv_2mortal(newSViv(c_array[i]))); } inline_stack_done; }` [download] Then create a ./XS-Wrap-0.42 source distro directory. Then create a ./build_src.pl that contains: `use warnings; use strict; use InlineX::C2XS qw(c2xs); my $options = {WRITE_PM => 1, WRITE_MAKEFILE_PL => 1, VERSION => '0.42', MANIF => 1, T => 1, EXPORT_OK_ALL => 1, EXPORT_TAGS_ALL => 'all', }; c2xs('XS::Wrap', 'XS::Wrap', './XS-Wrap-0.42', $options);` [download] Then run `perl build_src.pl` Then modify the files generated in ./XS-Wrap-0.42 as needed. This could all be further customised and further automated, as could your approach. Cheers, Rob	[reply] [d/l] [select]
Re^2: Wrapping a C shared library with Perl and XS by stevieb (Canon) on Mar 18, 2017 at 03:07 UTC
The hack from `main` to `XS::Wrap` was by hand within the XS file, yes. Tomorrow, I will review your feedback and play with it. Thanks as usual, syphilis. -stevieb update: I'm sure I've, in passing, had someone else point out to me how to modify these elements of the XS directives on the fly, but my searching of PM, the Internet and my scrambled brain couldn't pick up on it. That's essentially why my 'config' args to `Inline` look so ugly. I don't use it enough to remember :)	[reply] [d/l] [select]
Re^2: Wrapping a C shared library with Perl and XS by stevieb (Canon) on Mar 18, 2017 at 03:19 UTC
"I'd attack the problem using InlineX::C2XS, but few others seem to share my enthusiasm for that (with good reasons, no doubt)." Why is that?	[reply]
Re^3: Wrapping a C shared library with Perl and XS by syphilis (Archbishop) on Mar 18, 2017 at 05:45 UTC
Why is that ? Dunno - but I think along the lines of too kludgy or over-documented or under-documented or poorly documented. It's also lacking some features - eg choice between DynaLoader/XSLoader, and the $VERSION construct might not be to everyone's liking. (Plus more.) It suits me fine, so I'm not too stressed about it. Cheers, Rob	[reply]
Re: Wrapping a C shared library with Perl and XS by stevieb (Canon) on Mar 17, 2017 at 18:24 UTC
Here is the C code used for the above example, including build steps. The header file (`xswrap.h`) `int mult (int x, int y); void speak (const char* str); unsigned char* arr ();` [download] The .c file (`xswrap.c`) `#include <stdio.h> #include <stdlib.h> int mult (int x, int y){ return x * y; } void speak (const char* str){ printf("%s\n", str); } unsigned char* arr (){ unsigned char* list = malloc(sizeof(unsigned char) * 3); int i; for (i=0; i<3; i++){ list[i] = i; } return list; }` [download] The entry point file (`main.c`) for testing the lib `#include <stdio.h> #include "xswrap.h" int main (){ int ret = mult(5, 5); printf("%d\n", ret); speak("hello, world!"); unsigned char* list = arr(); int i; for (i=0; i<3; i++){ printf("%d\n", list[i]); } return 0; }` [download] The build/install script (`build.sh`) `#!/bin/sh gcc -c -fPIC xswrap.c gcc -shared -fPIC -Wl,-soname,libxswrap.so -o libxswrap.so xswrap.o -l +c sudo cp libxswrap.so /usr/lib sudo cp xswrap.h /usr/local/include` [download] Done! The library and its header file are both now put into directories in your PATH. To compile the test program: `gcc -o test main.c -lxswrap` [download] ...run it: `./test` [download] You're now ready to wrap the library using Perl and XS per the OP.	[reply] [d/l] [select]
Re: Wrapping a C shared library with Perl and XS by Anonymous Monk on Mar 28, 2017 at 16:21 UTC
It might be worth pointing out for other readers that after `arr()` returned a `malloc`ed array and you have handled its contents it's your XS code's obligation to `free()` it. Otherwise, this code would have a memory leak.	[reply] [d/l] [select]
Re^2: Wrapping a C shared library with Perl and XS by stevieb (Canon) on Mar 28, 2017 at 16:35 UTC
ohhh, right! I completely forgot about that :) Thanks for pointing it out!	[reply]
Re^2: Wrapping a C shared library with Perl and XS by stevieb (Canon) on Mar 29, 2017 at 03:31 UTC
It is honestly a bit elusive to me exactly how and when to `free()` within the XS in the case I've shown explicitly, but using the same repo I used to generate this example tutorial, I'm going to play around a bit here. I will post back here and update my OP once I figure it out. As always, feedback welcome ;)	[reply] [d/l]
Re^3: Wrapping a C shared library with Perl and XS by Anonymous Monk on Mar 31, 2017 at 20:11 UTC
One of the approaches to freeing memory is to call `free` right after the last usage of a pointer (in our case it's the loop which calls `inline_stack_push(sv_2mortal(newSViv(c_array[i])));`) and set it to `NULL` afterwards. Setting a pointer to NULL serves double purpose: Checking for NULL is very easy, checking if non-NULL pointer is valid is very complex at best If you defensively call `free(c_array)` in another part of the program, `free(NULL)` would be safe on modern `libc`s, while `free` of a `free`d pointer causes memory corruption	[reply] [d/l] [select]
Re^4: Wrapping a C shared library with Perl and XS by stevieb (Canon) on Mar 31, 2017 at 23:23 UTC
Re^5: Wrapping a C shared library with Perl and XS by Anonymous Monk on Apr 06, 2017 at 17:38 UTC
Some notes below your chosen depth have not been shown here


The stupid question is the question not asked
	PerlMonks