Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re^4: Perl crash during perl_clone

by perlmonk1729 (Acolyte)
on Oct 28, 2010 at 06:01 UTC ( [id://867918]=note: print w/replies, xml ) Need Help??


in reply to Re^3: Perl crash during perl_clone
in thread Perl crash during perl_clone

if the callback gets triggered at a point in teh application when the "wrong" interpreter has the cpu, then it fails

There are only two threads/interpreters and the thread that cloned an interpter does nothing else but wait for the right event to invoke the callback. So, bit confused what could be the 'wrong' interprter

The solution I'm afraid is to re-think your methodology. If you can describe why you are starting a second interpreter, we might be able to see a solution.

Infact, I did not know anything about interpters or cloning when I started this project. I learnt how to modify the SWIG generated wrapper to register the callback (RegisterCB in my example): I cache the perl sub reference and instead register a proxy C method (InvokeCB) into the C-api interface. When an unrelated thread (BGThread) called InvokeCB, it sets up the call-stack properly to call perl subroutines and then calls the sub via the cached reference.

However, the "call_sv" would always crashed - 100%. Then I stumbled across the perl_clone() and found many posts that claimed it was required because BGThread cannot call code into an interpreter it did not create (i.e perl.exe).

Having a cloned interpter allowed call_sv to work reliably..until my latest problems. You can see this link about my previous experiences that got me here. http://markmail.org/thread/cjbhybjfikuirvud

ps: I'm in asian timezone, so responses may seem to have a 'lag'.

Replies are listed 'Best First'.
Re^5: Perl crash during perl_clone
by BrowserUk (Patriarch) on Oct 28, 2010 at 08:23 UTC
    There are only two threads/interpreters and the thread that cloned an interpter does nothing else but wait for the right event to invoke the callback. So, bit confused what could be the 'wrong' interprter ... because BGThread cannot call code into an interpreter it did not create (i.e perl.exe).

    We agree that there are two interpreters. We agree that it is important that the "right" interpreter be used for the callback.

    The problem is, if the callback is triggered by an asynchronous event, how can you guarantee that only right interpreter is running when the callback occurs?

    Because if you don't take steps to ensure that, then it will inevitably happen that sometimes the triggering event will occur during the time-slice of the wrong interpreter, and it will segfault.

    Have you tried my suggestion of logging the current thread id from a) the place where you get the Perl callback address; b) where you invoke that Perl callback address?


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      I expected the thread-ids to be different by virtue of my design. Anycase, I logged the 'current' thread-id and confirmed that the id when I get the perl-callback address (in the SWIG wrapper) is different (it must be a thread created by perl.exe) from when the perl cback is invoked (the thread I created). No surprise.

      However, I got thinking about what you mean by the 'right' interpreter..if I have two interpreters then it is not clear what is the 'right' interpreter. Can you elaborate which you think is the right interpreter?

      Then I decided to question the premise/info under which I went to the cloning method and did an experiment: I removed the perl_clone completely (and the related SvSHARE, sv_dup etc). Instead, when my "BGThread-equivalent" is created, I just forced a PERL_SET_CONTEXT() to the interpreter created by perl.exe

      This forces the callback thread to invoke the perl sub in the same interpreter context as that created by perl.exe itself.

      In my limited tests, very surprisngly for me, so far that seems to work as well as the perl_clone approach I used for almost a yr now.

      I'll be testing more, but does this surprise you?

        Can you elaborate which you think is the right interpreter?

        The right interpreter, is whichever one the perl callback address originated in. That is, if your perl code--running in your main (perl.exe) thread, is the one that passes the address of a perl sub into your C code, then that is the right interpreter.

        Because, whilst call code and data--perl or C--is accessible to every thread at the C level, at the Perl source level, perl relies upon segregating memory and code on a per interpreter/thread basis. It does this for both internal reasons, and for those of simplifying the use of threads by imposing this "restricted view" on the programmer. That is the raison d'etre of the iThread model.

        To restate that another way; the reason Perl won't allow you to pass a coderef from one thread to another via shared memory, is because allowing you to do so would require imposing even greater performance penalties upon threads than currently. It would be necessary to apply internal locking to every perl subroutine, because they might be called concurrently on two or more threads.

        The approach iThreads uses to preventing both the programmer discipline that would be required were coderefs shareable (see the :lock keyword in the 5005 threads model), and the additional internal locking that would be required, is to require each interpreter/thread to only use subroutines compiled within that interpreter.

        Your use of C/XS is bypassing that protection mechanism--hence the crashes.

        I just forced a PERL_SET_CONTEXT() to the interpreter created by perl.exe.... but does this surprise you?

        No. By forcing the same context as was in use when the coderef was taken, you ensure that the code and data (closures etc.) behind that coderef is accessible. I would anticipate that, so long as your program remains single-threaded, that using this method to force the "correct context" for the callback will work reliably.

        If you wanted to ensure multi-threaded compatibility--then you can extend this mechanism by:

        • When your C/XS code receives the callback address from Perl, take a copy of the current context and save it away in a suitable place.
        • Later, when the triggering event occurs, save a copy of the then current context in a local var;
        • do a set_context() using the context you saved earlier;
        • call the coderef.
        • when the coderef returns; reset the current context.

        Note: All of the above is untested--by me; and probably by anyone else--so you are the pathfinder here. I'll willingly try and help with any problems that arise, but please understand that at this point, the only test-bed I have is my brain. And that is a poor substitute for a modern processor :)

        As an (interesting to me but probably not to you at this point) aside, this discussion leads to the possibility that could further reduce the overheads of the iThreads model. If coderefs explicitly carried their compile-time contexts with them, then it might be possible to allow a single copy of a Perl coderef--opcode tree--to be safely shared by multiple interpreters. But that's a thought to explore another time.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://867918]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others perusing the Monastery: (9)
As of 2024-03-29 15:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found