Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

How to setup the DynaLoader in a dynamically loaded perl?

by sciurius (Sexton)
on May 05, 2020 at 11:18 UTC ( [id://11116476]=perlquestion: print w/replies, xml ) Need Help??

sciurius has asked for the wisdom of the Perl Monks concerning the following question:

I can use the perl runtime from a C application using the recipes written in the documentation, e.g. (trivial lines left out for brevity): 
#include <EXTERN.h>
#include <perl.h>
#include "XSUB.h"

void boot_DynaLoader (pTHX_ CV* cv);

void xs_init(pTHX) {
    static const char file[] = __FILE__;
    dXSUB_SYS;
    PERL_UNUSED_CONTEXT;
    newXS( "DynaLoader::boot_DynaLoader", boot_DynaLoader, file );
}

int main( int argc, char **argv, char **env ) {
  PerlInterpreter *my_perl;
  PERL_SYS_INIT3( &argc, &argv, &env );
  my_perl = perl_alloc();
  perl_construct(my_perl);
  perl_parse( my_perl, xs_init, argc, argv, env );
  int result = perl_run(my_perl);
  perl_destruct(my_perl);
  perl_free(my_perl);
  PERL_SYS_TERM();
  return result;
}

[download]
This works great, however this requires that the application program is linked against the perl shared library. I want to distribute this application with a custom perl shared library. Still not a problem, I link the application with --rpath=. and place the shared library next to the application, for example: 
$ cd /tmp/foo
$ ls
myapp    myperl.so
$ ./myapp
Hello, world!

[download]
Still fine. But I want to be able to call the application from arbitrary places, e.g. 
$ cd
$ ls /tmp/foo
myapp    myperl.so
$ /tmp/foo/myapp
/tmp/foo/myapp: error while loading shared libraries: myperl.so: canno
+t open shared object file: No such file or directory

[download]
This is as it should be. When linking with --rpath=. the shared library must be in the current directory. What I would want is that the shared library is looked up there where the application is. I can use a wrapper script that sets LD_LIBRARY_PATH before calling the application but I'd rather want to solve this within the application itself. Fortunately, there is such a thing as dlopen: 
int main( int argc, char **argv, char **env ) {
  /* find path to shared library */
  /* open shared lib */
  void *handle = dlopen(...);

  /* Get entry point for perl_alloc */
  void* (*perl_alloc)();
  perl_alloc = (void*())dlsym(handle, "perl_alloc")
  /* Call perl_alloc */
  my_perl = (*perl_alloc)();

  /* and so on */
  /* ... */
}

[download]
This, too, works nice ... except for one detail: I cannot get the second argument to perl_parse (xs_init above, to set up the DynaLoader) right. Can anyone shed some light on how to get this working?

Replies are listed 'Best First'.
Re: How to setup the DynaLoader in a dynamically loaded perl?
by bliako (Monsignor) on May 05, 2020 at 14:46 UTC

    On second glance, you did not dlsym'ed boot_DynaLoader() that's why xs_init() had trouble.

    This is a complete, compilable and runnable example:

    /*
author: bliako
date:   05/05/2020
for:    https://perlmonks.org/?node_id=11116476

compile with:

$(perl -MConfig -e 'print $Config{cc}') a.c $(perl -MExtUtils::Embed -
+e ccopts -e ldopts) -o aaa -ldl

run with:

./aaa -e 'print "hello there, I am a tiny wee Perl!!\n"'

*/
#include <EXTERN.h>
#include <perl.h>
#include "XSUB.h"
#include <dlfcn.h>

// void boot_DynaLoader (pTHX_ CV* cv);
/* this declares a function pointer as opposed to an external function
+ above */
void (*boot_DynaLoader)(pTHX_ CV* cv);

void xs_init(pTHX) {
    static const char file[] = __FILE__;
    dXSUB_SYS;
    PERL_UNUSED_CONTEXT;
    newXS( "DynaLoader::boot_DynaLoader", boot_DynaLoader, file );
}

static PerlInterpreter *my_perl;
int main( int argc, char **argv, char **env ) {
    /* find path to shared library */
    /* open shared lib */
    void *handle = dlopen("/usr/lib64/libperl.so.5.28.2", RTLD_NOW);

    /* Get entry point for perl_alloc */
    void* (*perl_alloc)();
    perl_alloc = (void *(*)() )dlsym(handle, "perl_alloc");
        /* Get all the functions you will be using from the library, t
+his one is obvious */
    boot_DynaLoader = (void (*)(pTHX_ CV* cv) )dlsym(handle, "boot_Dyn
+aLoader");

    /* Call perl_alloc */
    my_perl = perl_alloc();

    /* and so on */
    perl_construct(my_perl);
    perl_parse( my_perl, xs_init, argc, argv, env );
    int result = perl_run(my_perl);
    perl_destruct(my_perl);
    perl_free(my_perl);
    PERL_SYS_TERM();
    return result;
}

    [download]

    Edit: added a typecast to boot_DynaLoader = (void (*)(pTHX_ CV* cv) )dlsym(handle, "boot_DynaLoader");

    bw, bliako

[reply]
[d/l]
[select]
      Thanks for your helpful suggestions. Your example compiles and runs, but: 
      $ ldd aaa
        linux-vdso.so.1 (0x00007ffd0e4c6000)
        libperl.so.5.30 => /lib64/libperl.so.5.30 (0x00007fe9264b5000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe926493000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fe92648c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fe9262c3000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fe92617d000)
        libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007fe926142000)

      [download]
      So it is already linked to the perl shared library. If I remove -lperl from the link arguments I get: 
      in function `xs_init':
/home/jv/tmp/aaa.c:28: undefined reference to `PL_thr_key'
/bin/ld: /home/jv/tmp/aaa.c:28: undefined reference to `pthread_getspe
+cific'
/bin/ld: /home/jv/tmp/aaa.c:28: undefined reference to `Perl_newXS'

      [download]
      EDIT: Managed to get it going by adding a Perl_newXS function as follows: 
      CV* Perl_newXS(pTHX_ const char *name, XSUBADDR_t subaddr, const char 
+*filename) {
  CV* (*imp)(tTHX, const char*, XSUBADDR_t, const char*);
  imp = (CV* (*)(tTHX, const char*, XSUBADDR_t, const char*)) dlsym( h
+andle, "Perl_newXS" );
  return (*imp)(my_perl, name, subaddr, filename);
}

      [download]
      The complete proof-of-concept program is now: 
      #include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>
#include <EXTERN.h>
#include <perl.h>

void *handle;            /* for dl */

static PerlInterpreter *my_perl;

void (*boot_DynaLoader)(pTHX_ CV* cv);

CV* Perl_newXS(pTHX_ const char *name, XSUBADDR_t subaddr, const char 
+*filename) {
  CV* (*imp)(tTHX, const char*, XSUBADDR_t, const char*);
  imp = (CV* (*)(tTHX, const char*, XSUBADDR_t, const char*)) dlsym( h
+andle, "Perl_newXS" );
  return (*imp)(my_perl, name, subaddr, filename);
}

void xs_init(pTHX) {
    static const char file[] = __FILE__;
    dXSUB_SYS;
    PERL_UNUSED_CONTEXT;
    newXS( "DynaLoader::boot_DynaLoader", boot_DynaLoader, file );
}

int main( int argc, char **argv, char **env ) {
  /* open shared lib */
  handle = dlopen("perl530.so", RTLD_LAZY);
  if ( !handle ) {
    fprintf( stderr, "%s\n", dlerror() );
    exit(EXIT_FAILURE);
  }

  /* Get entry point for perl_alloc */
  void* (*perl_alloc)();
  perl_alloc = (void*(*)()) dlsym(handle, "perl_alloc");
  /* Call perl_alloc */
  my_perl = (*perl_alloc)();

  /* perl_construct */
  void (*perl_construct)(void*);
  perl_construct = (void(*)(void*)) dlsym(handle, "perl_construct");
  (*perl_construct)(my_perl);

  /* boot_DynaLoader */
  boot_DynaLoader = (void (*)(pTHX_ CV* cv)) dlsym(handle, "boot_DynaL
+oader");
 
  /* perl_parse */
  void (*perl_parse)(void*, void*, int, char**, char**);
  perl_parse = (void(*)(void*, void*, int, char**, char**)) dlsym(hand
+le, "perl_parse");
  (*perl_parse)(my_perl, xs_init, argc, argv, env);

  /* perl_run */
  int (*perl_run)(void*);
  perl_run = (int(*)(void*)) dlsym(handle, "perl_run");
  int result = (*perl_run)(my_perl);

  /* perl_destruct */
  void(*perl_destruct)(void*);
  perl_destruct = (void(*)(void*)) dlsym(handle, "perl_destruct");
  (*perl_destruct)(my_perl);

  /* perl_free */
  void(*perl_free)(void*);
  perl_free = (void(*)(void*)) dlsym(handle, "perl_free");
  (*perl_free)(my_perl);

  return result;
}

      [download]
      Build with: $(perl -MConfig -e 'print $Config{cc}') p.c $(perl -MExtUtils::Embed -e ccopts ) -o aaa -ldl -Wl,--rpath=. Although it seems that dynamic loading is not yet functional... For example, this fails: 
      aaa -MFcntl=:flock -E "say 1+LOCK_NB"
Undefined subroutine &Fcntl::LOCK_NB called at -e line 1.

      [download]
[reply]
[d/l]
[select]

        As I said, you will need to dlsym anything that you may use later.

        However, when I did this:

        CV* (*Perl_newXS)(pTHX_ const char *name, XSUBADDR_t subaddr, const ch
+ar *filename);

....
Perl_newXS = (CV* (*)(pTHX_ const char *name, XSUBADDR_t subaddr, cons
+t char *filename) )dlsym(handle, "Perl_newXS");

        [download]

        I got

         error: ‘Perl_newXS’ redeclared as different kind of symbol
/usr/lib64/perl5/CORE/proto.h:2517:19: note: previous declaration of ‘
+Perl_newXS’ was here
 2517 | PERL_CALLCONV CV* Perl_newXS(pTHX_ const char *name, XSUBADDR_
+t subaddr, const char *filename);

        [download]

        So, that's a neat trick you have there! (though I would make imp static)

[reply]
[d/l]
[select]
Re: How to setup the DynaLoader in a dynamically loaded perl?
by bliako (Monsignor) on May 05, 2020 at 13:15 UTC

    What's the actual error in your last paragraph?

    Also, wouldn't setting rpath to an absolute path be less of a headache?

    Finally, creating a statically-linked executable, including a static link to perl and a static link to ALL the XS modules you may need sounds like a big hassle but once you set the workflow, you stop worrying about all the problems your binary may face in foreign machines.

    bw, bliako

[reply]

      Setting rpath to an absolute path would make things easier, but the application will not be installed in a predefined location.

      Creating a statically-linked executable would be an alternative. For the time being I'd prefer not to.

[reply]
Re: How to setup the DynaLoader in a dynamically loaded perl?
by Anonymous Monk on May 06, 2020 at 08:33 UTC
    When linking with --rpath=. the shared library must be in the current directory. What I would want is that the shared library is looked up there where the application is.
    man 8 ld.so:

    The dynamic linker understands certain token strings in an rpath specification (DT_RPATH or DT_RUNPATH). Those strings are substituted as follows:

    • $ORIGIN (or equivalently ${ORIGIN})

      This expands to the directory containing the program or shared object. Thus, an application located in somedir/app could be compiled with 

                        gcc -Wl,-rpath,'$ORIGIN/../lib'

      [download]
      so that it finds an associated shared object in somedir/lib no matter where somedir is located in the directory hierarchy.

[reply]
[d/l]
[select]

      Fantastic! That's exactly what I'm looking for.

      (Interesting: in "man ld" $ORIGIN is only mentioned in the context of --rpath-link so I didn't think this was appliccable in this case. Thank you for pointing out.)

[reply]
Re: How to setup the DynaLoader in a dynamically loaded perl?
by Anonymous Monk on May 06, 2020 at 00:56 UTC

    I can use the perl runtime from a C application using the recipes written in the documentation, e.g. (trivial lines left out for brevity):

    What documentation?

    I've never seen or needed that DynaLoader stuff to embed a perl

[reply]
      https://perldoc.perl.org/perlembed.html#Using-Perl-modules%2c-which-themselves-use-C-libraries%2c-from-your-C-program

      If I don't setup the DynaLoader like this I get errors like this (IRL, not in the sample): 

      Can't load module Encode, dynamic loading not available in this perl.
  (You may need to build a new perl executable which either supports
  dynamic loading or has the Encode module statically linked into it.)
 at Encode.pm line 13.

      [download]
[reply]
[d/l]
        Finally got it all up and running.

        One of the additional steps that were needed was to rename the so-name of the library in all .so files.

        For this I used the patchelf tool: 

            cp -pL ${PERLLIB}/libperl.so ${DEST}/${PERLSO}
    cp -pL ${PERLLIB}/libpthread.so.0 ${DEST}/
    patchelf --set-soname perl530.so ${DEST}/${PERLSO}
    find ${DEST} -name '*.so' -exec patchelf --replace-needed libperl.
+so.5.30 ${PERLSO} {} \;

        [download]
        The resultant, and working, program is now: 
        #include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>
#include <EXTERN.h>
#include <perl.h>

void *handle;            /* for dl */

#ifndef PERLSO
#  define PERLSO "perl530.so"
#endif

static pTHX;

void xs_init(pTHX);

int main( int argc, char **argv, char **env ) {

  /* Assuming the program binary   /foo/bar/blech */
  char selfpath[PATH_MAX];    /* /foo/bar */
  char scriptname[PATH_MAX];    /* blech */
  char dllpath[PATH_MAX];    /* /foo/bar/PERLSO */
  memset (selfpath,   0, PATH_MAX);
  memset (scriptname, 0, PATH_MAX);
  memset (dllpath,    0, PATH_MAX);

  if ( readlink ("/proc/self/exe", selfpath, PATH_MAX-1 ) > 0 ) {
    char *p = rindex( selfpath, '/' );
    if ( p ) {
      p++;
      strcpy( scriptname, p );
      *p = 0;
    }
    else
      strcpy( scriptname, selfpath );
#ifdef DEBUG
    fprintf( stderr, "selfpath:   %s\n", selfpath );
    fprintf( stderr, "scriptname: %s\n", scriptname );
#endif
  }
  else {
    strncpy( selfpath, argv[0], PATH_MAX-1 );
    char *p = rindex( selfpath, '/' );
    if ( p ) {
      p++;
      strcpy( scriptname, p );
      *p = 0;
    }
    else {
      p = getcwd( selfpath, PATH_MAX-1 );
      strcat( selfpath, "/" );
      strncpy( scriptname, argv[0], PATH_MAX-1 );
    }
#ifdef DEBUG
    fprintf( stderr, "cwdpath:    %s\n", selfpath );
    fprintf( stderr, "scriptname: %s\n", scriptname );
#endif
  }

  /* Open shared lib. */
  strcpy( dllpath, selfpath );
  strcat( dllpath, PERLSO );

#ifdef DEBUG
  fprintf( stderr, "dllpath:    %s\n", dllpath );
#endif

  handle = dlopen( dllpath, RTLD_LAZY );
  if ( !handle ) {
    fprintf( stderr, "%s\n", dlerror() );
    exit(EXIT_FAILURE);
  }

  /* Start perl environment. */
  //PERL_SYS_INIT3( &argc, &argv, &env );
  void (*Perl_sys_init3)( int*, char***, char*** );
  Perl_sys_init3 = (void(*)(int*, char***, char***)) dlsym(handle, "Pe
+rl_sys_init3");
  (*Perl_sys_init3)(&argc, &argv, &env);

  /* Create a perl interpreter. */
  tTHX (*perl_alloc)();
  perl_alloc = (tTHX(*)()) dlsym(handle, "perl_alloc");
  my_perl = (*perl_alloc)();

  /* perl_construct */
  void (*perl_construct)(pTHX);
  perl_construct = (void(*)(pTHX)) dlsym(handle, "perl_construct");
  (*perl_construct)(aTHX);

  /* perl_parse */
  void (*perl_parse)(pTHX, void*, int, char**, char**);
  perl_parse = (void(*)(pTHX, void*, int, char**, char**)) dlsym(handl
+e, "perl_parse");

  /* If we're "perl", behave like perl. */
  if ( !strncmp( scriptname, "perl", 4 ) )
    (*perl_parse)( aTHX, xs_init, argc, argv, env );

  else {

    /* Insert script name in argv. */
    char scriptpath[PATH_MAX];    /* /foo/bar/SCRIPTPREFIXblech.pl */
    strcpy( scriptpath, selfpath );
#ifdef SCRIPTPREFIX
    strcat( scriptpath, "script/" );
#endif
    strcat( scriptpath, scriptname );
    strcat( scriptpath, ".pl" );

    /* To get @INC right we execute it as a -E script. */
    char cmd[2*PATH_MAX+100];
    sprintf( cmd, "@INC=(q{%slib});do q{%s};", selfpath, scriptpath );
#ifdef DEBUG
    fprintf( stderr, "scriptpath: %s\n", scriptpath );
    fprintf( stderr, "cmd:        %s\n", cmd );
#endif

#   define EXTRA_ARGS 3    
    char **ourargv = (char **)calloc( argc+1+EXTRA_ARGS, sizeof(char**
+) );
    ourargv[0] = argv[0];
    ourargv[1] = "-E";
    ourargv[2] = cmd;
    ourargv[3] = "--";
    for ( int i=1; i<=argc; ++i ) {
      ourargv[i+EXTRA_ARGS] = argv[i];
    }
    (*perl_parse)(aTHX, xs_init, argc+EXTRA_ARGS, ourargv, env);
  }

  /* Set @INC to just our stuff. But it's too late. */
  //  char cmd[PATH_MAX+100];
  //  sprintf( cmd, "@INC = (q{%slib});", selfpath );
  //  SV* (*eval_pv)(pTHX, const char*, I32);
  //  eval_pv = (SV* (*)(pTHX, const char*, I32)) dlsym( handle, "Perl
+_eval_pv" );
  //  (*eval_pv)( aTHX, cmd, TRUE );

  /* Run... */
  int (*perl_run)(pTHX);
  perl_run = (int(*)(pTHX)) dlsym(handle, "perl_run");
  int result = (*perl_run)(aTHX);

  /* Cleanup. */
  void(*perl_destruct)(pTHX);
  perl_destruct = (void(*)(pTHX)) dlsym(handle, "perl_destruct");
  (*perl_destruct)(aTHX);
  void(*perl_free)(pTHX);
  perl_free = (void(*)(pTHX)) dlsym(handle, "perl_free");
  (*perl_free)(aTHX);

  return result;
}

void (*boot_DynaLoader_dyn)(pTHX, CV* cv);
CV* (*Perl_newXS_dyn)(pTHX, const char*, XSUBADDR_t, const char*);

void xs_init(pTHX) {
    static const char file[] = __FILE__;
    // dXSUB_SYS;            /* dNOOP */
    // PERL_UNUSED_CONTEXT;

    /* boot_DynaLoader */
    boot_DynaLoader_dyn = (void (*)(pTHX, CV* cv)) dlsym(handle, "boot
+_DynaLoader");
    if ( !boot_DynaLoader_dyn ) {
      fprintf( stderr, "(boot_DynaLoader) %s\n", dlerror() );
      exit(EXIT_FAILURE);
    }

    /* newXS is just Perl_newXS(aTHX, ...) */
    Perl_newXS_dyn = (CV* (*)(pTHX, const char*, XSUBADDR_t, const cha
+r*)) dlsym( handle, "Perl_newXS" );
 
    /* The following comment is mandatory. */
    /* DynaLoader is a special case */
    (*Perl_newXS_dyn)( aTHX, "DynaLoader::boot_DynaLoader", *boot_Dyna
+Loader_dyn, file );
}

        [download]

        Thanks for your help!

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: perlquestion [id://11116476]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others rifling through the Monastery: (4)
As of 2024-04-19 17:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found