My advice is to not mix fork and threads. Use one or the other.

Especially on Windows, where fork is emulated by threads, mixing the two gives even weirder results.

Sorry if my post was unclear, but I didn't mix fork and threads in my program. I just tried both versions.

By the way, it's best to avoid fork on Windows if you have a choice. It's not really something Windows supports, so it's (badly) emulated using threads by Perl.

As to why your code fails? I suspect the use of an XS module that doesn't support multithreading.

>As to why your code fails? I suspect the use of an XS module that doesn't support multithreading.

I am not confident that this is correct. Based on my understanding if ithreads, it's not traditional light weight threads (e.g., pthreads), so there is no possibility of the threadsafeness of the shared library being tested. The ideal of threadded shared libraries is certainly interesting and I think appropriate if implemented carefully in the library with mindfulness that it's going to be used by a single perl parent process and perl datastructures are in no way thread safe.

There seems to be a severe lack of familiarity in the Perl community with what "real" threads are wrt "multithreading". No doubt this is not lost of many, but I continue to see over and over again the suggestion that Perl ithreads are related to traditional shared memory threads. This is simply not the case.

The suggestion that Perl ithreads should not be used on Windows also seems to contradict the author of Coro's explaination (in POD of Coro) of what Perl threads are, and based on my knowledge of threads he "gets it". In fact, he indicates that ithreads is actually appropriate to some degree for Windows but completely brain dead for *nix/POSIX platforms.

I offer the above simply to prevent the spread of false information, I am open to being wrong and would appreciate constructive correction in any form.

I can't make heads or tails of anything you are saying. These three points should handle it all, though:

• Perl uses real threads (pthread_create where available).

• Thread-safe XS libraries are possible, and many are. But supporting multi-threading may require extra work. (This isn't specific to XS.) For example, JSON::XS isn't thread safe (because its author intentionally refuses to support them on principle), but Cpanel::JSON::XS is.

• There's nothing wrong with using threads in unix.

• If you want to perform aynchronous operations, Coro is one of many useful approaches. Just beware that if any of your Coro threads use something that's incompatible with the event loop being used by Coro, everything will block since it's a co-operative multi-tasking system.

• If you want parallel processing, you'll need OS-level tasks (e.g. threads or processes), not what Coro provides.

Update: I meant "that's incompatible". Fixed.

>>As to why your code fails? I suspect the use of an XS module that doesn't support multithreading.

>I am not confident that this is correct. Based on my understanding if ithreads, it's not traditional light weight threads (e.g., pthreads), so there is no possibility of the threadsafeness of the shared library being tested. ...

From the threads documentation:

Since Perl 5.8, thread programming has been available using a model called *interpreter threads* which provides a new Perl interpreter for each thread, and, by default, results in no data or state information being shared between threads.

So, perl threads are just a thin layer over the C interface to the OS implementation of threads, with the heavyweight copy-everything step added. There's nothing I see there that would cause a previously thread-safe library to no longer be thread safe.

>There seems to be a severe lack of familiarity in the Perl community with what "real" threads are wrt "multithreading". No doubt this is not lost of many, but I continue to see over and over again the suggestion that Perl ithreads are related to traditional shared memory threads. This is simply not the case.

Lack of familiarity with the concept of "threads" or lack of familiarity with a particular implementation of "threads"? The concept of threads is easy, but threads are hard to use for everyone, unless someone has already abstracted it away for them.

First, there is no fork(2) call on Windows, so (as other monks have mentioned) Windows perls implement fork using interpreter threads. The obvious problem here is that creating a thread or forking a subprocess completes immediately and the main thread continues, then exits at the end of the script while the children are waiting at sleep 300. When the main thread exits the process, Windows kills the other threads.

On Linux, threads are implemented in terms of clone(2), which is a fork-like call, (NPTL added new system calls with different names long ago but has very similar semantics) so threads are actually lightweight processes. (The Linux kernel calls them both "tasks" instead of distinguishing threads and processes.) Again, both programs will do the same (different) thing: the original process spawns the others and exits, but under Linux, the created threads are more like forked processes, so either they continue to run after the parent exits or the kernel blocks the parent at exit(2) until the other threads complete.

For a solution, try either wait/waitpid or the ->join() method in threads instead of detaching your worker threads.

The obvious problem here is that creating a thread or forking a subprocess completes immediately and the main thread continues, then exits at the end of the script while the children are waiting at sleep 300. When the main thread exits the process, Windows kills the other threads.

The main thread didn't continue because the part of the code I showed was running in an indefinite loop. And nothing in the child processes or in the worker threads was executed either.

For a solution, try either wait/waitpid or the ->join() method in threads instead of detaching your worker threads.

I think I tried threads without detaching them. And since the part of the program is in an indefinite loop, I don't want to wait the processes to finish because the loop needed to be fast.

is sleep 300; honoured?

No, I tried adding say 1 after fork() and threads->create() and it didn't print it.

But now with Strawberry 32bit version it seems to work, this is bizarre. All I did differently is I installed WWW::Mechanize, which I use, with cpan. And it came with Strawberry built-in. At least perl -MWWW::Mechanize "print 1" printed 1. I don't understand at all what's happening. Things don't seem to work consistently at all.

So you say that it does/did not fork() at all? Have you investigated with a simple test script to see if forks happen, how many and under what conditions with what system messages? Can the OS limit the number of forks? Note that Windows does not have a native fork() and so Perl emulates that somehow. See perlfork for the caveats and warnings. (I am exclusively working with Linux so I can't help with anything else)

If anyone is interested what happened with this. I told my client to install 32bit Strawberry and gave him my program using forking and lo and behold; it worked on his machine! The version with threads exited like on my machine. Btw, I'm using Windows in Virtualbox, maybe it could be messing things up? All and all, this experience doesn't really make me confident to use Perl on Windows at all with these bizarre bugs. The program worked without any hiccups on Linux where I developed it. Thanks for all who replied.

Hi fluks,

The following are modified threads and fork demonstrations. They work on Linux including 32-bit and 64-bit Windows.

For threads, one may need to increase stack_size depending on the application. Not reaping threads will likely cause recent threads to not complete and exit prematurely. Ditto for fork on the Windows platform. Tip: Loading IO::Handle before spawning workers is beneficial. This saves workers from having to load this particular module and dependencies individually.

threads

use strict;
use warnings;

use threads (stack_size => 64*4096);
use IO::Handle (); # extra stability

my @car_links = (1..200);
my ($username, $password, $fh, @known_cars);

sub reserve_car { }
sub write_url_to_known_cars { }

my $count = 0;

for my $c (@car_links) {
    CREATE:
    my $thr = threads->create(sub {
        # Reserve for 10 seconds total.
        for (1..1) {
            reserve_car($c, $username, $password);
            sleep 10;
        }
        threads->exit(0);
    });
    if (!defined $thr) {
        while () {
            $count++;
            warn "cannot spawn thread, waiting -- $count\n";
            my $thr_exited = 0;
            for my $thr (threads->list(threads::joinable)) {
                $thr->join;
                $thr_exited = 1;
            }
            $thr_exited ? last : sleep(1);
        }
        goto CREATE;
    }

    push @known_cars, $c;
    write_url_to_known_cars($c, $fh);
}

print "reaping threads\n";
$_->join for threads->list;
[download]

fork

use strict;
use warnings;

use POSIX ":sys_wait_h";
use IO::Handle (); # extra stability

my @car_links = (1..100);
my ($username, $password, $fh, @known_cars);

sub reserve_car { }
sub write_url_to_known_cars { }

my $count = 0;
my %pids;

for my $c (@car_links) {
    FORK:
    my $pid = fork();
    if (!defined $pid) {
        while () {
            $count++;
            warn "cannot spawn child, waiting -- $count\n";
            my $child_exited = 0;
            for my $pid (keys %pids) {
                # support negative PID value on Windows
                my $ret = waitpid($pid, WNOHANG);
                if ($ret < -1 || $ret > 0) {
                    delete $pids{$pid};
                    $child_exited = 1;
                }
            }
            $child_exited ? last : sleep(1);
        }
        goto FORK;
    }
    elsif ($pid == 0) {
        # Reserve for 10 seconds total.
        for (1..1) {
            reserve_car($c, $username, $password);
            sleep 10;
        }
        exit;
    }
    else {
        $pids{$pid} = undef;
    }

    push @known_cars, $c;
    write_url_to_known_cars($c, $fh);
}

print "reaping children\n";
waitpid($_, 0) for (keys %pids);
[download]

Regards, Mario

If you're merely looking for a better programming interface to fork so that you're not tempted to mix "threads" with "fork", then take a look at Parallel::ForkManager.

If you need low latency communication among child processes or between parent/child (via fork), use something like redis as the middleware. It is unfortunate communication using shared memory isn't possible natively in perl, but that's a whole other issue.