Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Passing globs between threads

by conrad (Beadle)
on Sep 30, 2004 at 15:14 UTC ( [id://395373]=perlquestion: print w/replies, xml ) Need Help??

conrad has asked for the wisdom of the Perl Monks concerning the following question:

I'm looking for the best way to transmit file/socket/pipe handles between threads. I've been using Storable to communicate jobs to a worker thread (by freezing the job data structure into a shared queue which the thread then pops and thaws it from - is this regarded as Good Practice?), but since Storable can't serialise globs, I can't pass the thread a new IO::Handle to work with.

My first reaction was to try to locally add STORABLE_freeze and STORABLE_thaw methods to IO::Handle, but this doesn't work since IO::Handle objects are actually blessed globs, and (looking at the .xs source) Storable dies (because of its problematic underlying type) before I get the opportunity to use my own "smart" serialising code on the blessed glob.

Beginning to think my only option is to break IO::Handles down to tuples containing something like

  • class
  • file descriptor
  • open mode

.. and recreate using $class->new_from_fd($fd, $mode), but here I'm stumped as to how to work out what the opening mode of the thing is just from its glob.

Any suggestions please?

Addition #1: I've added example code below.

Addition #2: I'm actually not stumped by the open mode thing, was just being lazy - but is this the best way to do things?

Replies are listed 'Best First'.
Re: Passing globs between threads
by BrowserUk (Patriarch) on Sep 30, 2004 at 16:29 UTC

    This works fine for me.

    #! perl -slw use strict; use IO::File; use threads; sub thread{ my $fh = shift; print for <$fh>; return; } my $io = new IO::File( 'junk', 'r' ) or die $!; my $thread = threads->create( \&thread, $io ); $thread->join; $io->close; __END__ P:\test>395373 This is junk and some more junk and yet more junk this is line 4 of junk

    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
    "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
      Sorry, should have submitted example code, I wasn't being clear: I'm passing objects to the thread through shared variables after it's been created. This code demonstrates what I mean - note that it won't complete successfully as long as the *STDIN line remains uncommented. But I'm looking for the best way of making something like that work!

      #!/usr/bin/perl -w use strict; use threads; use threads::shared; use IO::Handle; use Thread::Semaphore; # Here we've a choice - use Storable to freeze and thaw stuff, or # these two bogus implementations which do nothing. This illustrates # (when you use the bogus code) how you cannot simply drop blessed # objects into shared objects. If you use Storable, blessed objects # can be passed across while they're frozen, but file handles (for # example) still can't. use Storable qw{freeze thaw}; #sub freeze { ${$_[0]} } #sub thaw { \shift } # Job queue and semaphore; the semaphore gets signalled every time an # object is pushed onto the queue. my @queue : shared; my $queue_ctrl = Thread::Semaphore->new(0); sub thread { while(1) { $queue_ctrl->down; # Wait for something to be queued lock @queue; last unless @queue; # Terminate on empty packet my $object = ${ thaw shift @queue }; # Would normally receive something relatively complex and do some # work with it here. print "Thread received $object.\n"; } } # Note that I'm creating the thread *first* and *afterwards* passing # objects across (through the queue) for it to work with.. my $thr = threads->new(\&thread); # Things we want to try passing through to the thread: my @stuff = ( # A basic type, works no problem with or without freeze/thaw '"hi there"', # A blessed reference, must be frozen and thawed bless({}, 'Foo'), # We can pass STDIN's file descriptor as an integer STDIN->fileno, # Have to comment this next line out for successful completion - but # this (or some equivalent) is what I would like to achieve! *STDIN, ); # Queue stuff and see what happens foreach my $obj ( @stuff ) { { # Queue $obj and signal $queue_ctrl lock @queue; print "\nEnqueueing $obj...\n"; push @queue, freeze \$obj; $queue_ctrl->up; } sleep 1; # Optional, but gives the thread a chance to grab the queue } # Terminate - signal $queue_ctrl without enqueueing anything. $queue_ctrl->up; $thr->join;

        The first thing to realise is that raw filehandles and sockets are process global. So you don't need, and indeed cannot, share them between threads in the threads::shared sense, as they are effectively already shared by all threads in the process.

        Note: I'm talking about the OS & C-runtime concept of filehandles and sockets, not anything in the IO::* group of modules. These are peculiar beasts in that they are at some level like ordinary perl objects, which makes them difficult, if not impossible to use safely across threads, but they do not behave entirely like ordinary objects.

        So, the problem is how to transfer an open handle between threads (which at a higher level seems like a bad design to me, but I'll get back to that), given that threads::shared won't let you share a GLOB nor even a 5.8.x lexical scalar that is currently being used as a GLOB-like entity.

        After a little kicking around, I found a way. I'm not yet sure that it is a good thing to do, but I'll show you how anyway and hope that if someone else out there knows why it should not be done, they'll speak up.

        The trick is to pass the fileno of the open GLOB to the thread and then use the special form of open open FH, "&=$fileno" or die $! to re-open the handle within the thread before using it.

        #! perl -slw use strict; use IO::File; use threads; use Thread::Queue; sub thread{ my $Q = shift; my $fno = $Q->dequeue; open FH, "<&=$fno" or die $!; print for <FH>; return; } my $Q = new Thread::Queue; my $thread = threads->create( \&thread, $Q ); my $io = IO::File->new( 'junk', 'r' ) or die $!; $Q->enqueue( fileno $io ); $thread->join; $io->close; __END__ P:\test>395373 This is junk and some more junk and yet more junk this is line 4 of junk

        However, this is not a complete solution. Currently, probably because of a bug I think, but possibly by design, if you actually read from the handle in the thread where you opened it, then pass the fileno to another thread, reopen it and attempt to read some more from it, it fails. No errors, no warnings. Just no input.

        I've read and re-read perlfunc:open and perlopentut:Re-Opening-Files-(dups) , and I believe that the "&=$fileno" syntax should allow this...but it doesn't. If you need to be able to do this, you'll need to take it up with p5p guys and get their wisdom.

        Now, getting back to design. Whilst moving objects around between threads by freezing and thawing them is possible, it feels awefully hooky to me. Basically, if your design is done correctly, there should be no need to create duplicates of objects in different threads.

        Doing so is opening yourself up to a world of greif.

        These duplicate objects will be uncoordinated. Once you freeze an object, it is exactly that; frozen. Any subsequent changes you make will not be reflected in the copy that you reconstruct elsewhere. If it was trivial, or even if it was possible with a reasonable degree of difficulty to coordinate and synchronise objects across threads, Perl would do that for you. The fact that the clever guys that got iThreads this far did not step up to the plate and do this already, almost certainly means that you should not be trying to do this either.

        The fact is that I appear to have done as much coding of(i)threads as anyone I am aware of, and I have yet to see a usefully iThreadable problem I couldn't (almost trivially) solve. And so far, I have never needed to share either objects (or IO handles) between threads.

        But don't take my word for it. My knowledge only extends as far as I have tried to go. It would be good to see someone else pushing the boundaries of what's possible.

        If you could describe the problem that you are trying to solve at the application level, I'd be most interested to take a look.


        Examine what is said, not who speaks.
        "Efficiency is intelligent laziness." -David Dunham
        "Think for yourself!" - Abigail
        "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://395373]
Front-paged by diotalevi
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (3)
As of 2024-04-25 14:48 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found