XS replacing my c library, not compatible with OS threads

patcat88 has asked for the wisdom of the Perl Monks concerning the following question:

I'm writing an XS module and Perl seems to be rewriting my C standard library from Visual Studio CRT on win32. malloc() free() and exit() are being redefined to perl specific versions, which can not work inside the thread. Why is Perl rewriting my C standard library functions? I know Perl has its internal C library from perlclib, so why are my C library's functions being redefined? Perl functions never work outside the Perl thread (no "context"), so the moment I try using a C library function in the other thread, they crash. The free() will always crash. The code sample is not compilable. Its shortened from something much bigger.

#define  WIN32_LEAN_AND_MEAN
#include <windows.h>
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

DWORD WINAPI ThreadFunc ( HANDLE * eventPtr) {
     DWORD retval;
     retval = WaitForSingleObject(*eventPtr, 10000);
     if(retval == WAIT_OBJECT_0) {
     printf("wait suceeded");
     }
     else if(retval == WAIT_TIMEOUT){
          //crash gracefully
          printf("Timeout, now exiting"); exit(1);}
     }
     //crash gracefully
     else {printf("WaitForSingleObject failed"); exit(1);}
     if(!CloseHandle(*eventPtr) {
          //crash gracefully
          printf("CloseHandle failed error is %u\n", GetLastError()); 
+exit(1);
     }
     free(eventPtr);
     return 1;
}

MODULE = MyMod    PACKAGE = MyMod

PROTOTYPES: DISABLE

DWORD
perlfunc ()
PREINIT:
HANDLE  *   evtHandlePtr;
HANDLE     thHandle;
CODE:
evtHandlePtr = malloc(sizeof(HANDLE));
*evtHandlePtr = CreateEvent(NULL, TRUE, FALSE, NULL);
if(*evtHandlePtr == NULL){printf("CreateEvent failed error is %u\n",Ge
+tLastError()); exit(1);}
thHandle = CreateThread(NULL, 1024, ThreadFunc, evtHandlePtr, 0, NULL)
+;
if(thHandle == NULL){printf("CreateThread failed error is %u\n",GetLas
+tError()); exit(1);}
else {
     if (!CloseHandle(thHandle))
     {printf("CloseHandle on thread handle failed error is %u\n",GetLa
+stError()); exit(1);}
}
//lets time out, so no SetEvent
RETVAL = 1;
OUTPUT:
     RETVAL
[download]

"exit(1);" becomes "(*(*Perl_IProc_ptr(((PerlInterpreter *)Perl_get_context())))->pExit)((*Perl_IProc_ptr(((PerlInterpreter *)Perl_get_context()))), (1));" everywhere

printf stays the same everywhere

"free(eventPtr);" becomes "(*(*Perl_IMem_ptr(((PerlInterpreter *)Perl_get_context())))->pFree)((*Perl_IMem_ptr(((PerlInterpreter *)Perl_get_context()))), (eventPtr));"

"evtHandlePtr = malloc(sizeof(HANDLE));" becomes "evtHandlePtr = (*(*Perl_IMem_ptr(((PerlInterpreter *)Perl_get_context())))->pMalloc)((*Perl_IMem_ptr(((PerlInterpreter *)Perl_get_context()))), (sizeof(HANDLE)));"

I need to be able to allocate memory inside the XS func, then eventually free it from the thread. The ultimate goal is a timeout feature that creates a thread to cancel a blocking function that is called in the XS func. The full timeout code inside the thread crashes on the free() always. The blocking function sometimes might never return hanging forever due to bugs in the that closed source library. The developer of the library suggests making another thread to end the job if a timeout period elapses without the blocking function returning (and then my XS function telling the timeout thread not to kill the job through a Win32 Event object).

But how can I use my native OS (windows)'s C library when Perl is redefining it?

The exit(1)s are supposed to be basically graceful crashes, but they would crash really hard if they ran inside the thread since perl doesn't exist inside that thread. I'm not sure if Perl's exit replacement will continue to execute destroyers and more perl code or not.

Also this closed source library I'm trying to wrap into XS, has errata that it will overrun the buffer you gave it in rare cases, and it returns an error code that it overran its buffer and you need to exit now since your stack and/or heap are corrupted.

How can I get a hard exit that is guaranteed not to run any more Perl code, or is basically a kill on the process?

Comment on XS replacing my c library, not compatible with OS threads Download Code

Replies are listed 'Best First'.
Re: XS replacing my c library, not compatible with OS threads by BrowserUk (Patriarch) on Nov 30, 2010 at 19:26 UTC
It's a right royal pain isn't it! Perhaps the simplest thing to do is compile your C code to a DLL & lib without any mention of Perl or XS. Ie. No perl or XS headers or makefile stuff. Then use h2xs to generate the XS interface wrappers. This means that your C functions would have to make no references at all to Perl internal APIs or data structures. So, no SV* parameters. It also means that you need to take full responsibility for freeing any dynamically allocated memory. That means providing destructors and ensuring that the get called in a timely fashion. It is possible to `#undef malloc` etc. in order to gain access back to the CRT versions; but you have to be very careful about restricting the scope of those undefs. It is all together a mess, and the primary reason many of my projects have never come to friution. So much of the library code I've written works perfectly when called from simple or threaded C-only test programs, but as soon as I try to call it from Perl, it just fails in random and mysterious ways and is neigh impossible to debug. It obviously is possible to do this stuff properly--there are plenty of Win32-only and ported XS modules that work fine--but the secrets of making it work seem to be mostly lost. All together intensely frustrating. Sorry I don't have better advice. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP an inspiration; A true Folk's Guy	[reply] [d/l]
Re: XS replacing my c library, not compatible with OS threads by SankoR (Prior) on Nov 30, 2010 at 21:17 UTC
Yep. The default behavior is great for core perl development but can be a major headache for green XS and embedding authors. The 'secret' is to `#define NO_XSLOCKS` before including XSUB.h. If left undefined, XSUB.h will redefine a long list of functions/macros (including free, malloc, and exit) with internal versions.	[reply] [d/l]
Re: XS replacing my c library, not compatible with OS threads by Marshall (Canon) on Dec 01, 2010 at 08:59 UTC
There can definitely be trouble if different threads are linking with different memory managers - there are some "tricks"! I stumbled across a couple points while working on a C/C++ project a few months ago - might or might not help you. malloc() free() and exit(), etc. live in MSVCRT.dll. That dll is the Window OS'es idea of these standard functions. That dll exists on every Windows system. VC2006 and gcc will link with that library. I guess somewhere along the way the OS guys and the compiler guys diverged in what they were doing and now there are many versions of MSVCRT that might or might not be on a particular Windows system. If for example, you have Visual Studio 2008, then that compiler will by default link with MSVCR90.dll a compiler specific library, not with MSVCRT.DLL. I don't know what flavor of Perl you are running or how it was built. A binary distribution is going to link with the OS's memory mgmt because who knows whether MSVCRXX will wind up being there or not. Anyway some of your woes may be related to this issue.	[reply]
Re^2: XS replacing my c library, not compatible with OS threads by BrowserUk (Patriarch) on Dec 01, 2010 at 10:24 UTC
Whilst mismatched C runtime libraries is certainly an issue with building XS extensions for use with binary distributions, it is far from being the predominant problem. The single biggest problem is trying to work out what actually gets called when you use an alloc call. There are so many #defines, #undefs, wrapper-functions and dispatch tables that conditionally define and redefine each of these calls, that you need to be a compiler to work out what actually gets called. And with so many levels of indirection and macro redefinition, setting breakpoints within a debugger is almost impossible. And as the OP indicated, even if you use the compiler /E switch (or equivalent) to view the post preprocessor output, what you get is so complicated--a simple `char p = malloc( 10 );` decoding to several hundred bytes of tortuously nested, and often duplicated) function calls, casts and parens--that it is still essentially unintelligible. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]
Re^3: XS replacing my c library, not compatible with OS threads by Corion (Patriarch) on Dec 01, 2010 at 10:28 UTC
There is at least a small utility distributed with Perl to help expanding the macros without having to talk to `gcc /E` yourself: expand-macro.pl. Of course that only shows you what the macros expand to, eventually, and not what the C code means.	[reply] [d/l]
Re^4: XS replacing my c library, not compatible with OS threads by BrowserUk (Patriarch) on Dec 01, 2010 at 11:08 UTC
Re: XS replacing my c library, not compatible with OS threads by cdarke (Prior) on Dec 01, 2010 at 09:26 UTC
Rather than use malloc and free, you could use the old Windows GlobalAlloc or LocalAlloc (with GlobalFree/LocalFree), or maybe even VirtualAlloc (which allocates pages).	[reply]


Problems? Is your data what you think it is?
	PerlMonks