I think libct isn't the XS component of the Perl module, but rather a C library used by the former. use lib won't help.

I suspect the problem lies with LD_LIBRARY_PATH. If so, your attempt to set it only executes after the module has already been loaded. Try

BEGIN { $ENV{LD_LIBRARY_PATH} = '...'; }
[download]

For some people, even that was too late. They had to do something like

BEGIN {
   if (!$ENV{LD_LIBRARY_PATH}) {
      $ENV{LD_LIBRARY_PATH} = '...';
      exec($^X, $0, @ARGV);
   }
}
[download]