Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

fork() on win10

by BernieC (Pilgrim)
on Jan 03, 2023 at 15:24 UTC ( #11149335=perlquestion: print w/replies, xml ) Need Help??

BernieC has asked for the wisdom of the Perl Monks concerning the following question:

I have a program that could benefit from being parallel processed, and it ought to be possible with the six-core processor on my win10 system. BUT: I read the caveats about fork on windows and I just confirmed that it is true: a simple fork doesn't work.
D:\Desktop\fork>cpu one process: 0:24 two processes: 0:53
Is there some trick or something with which I could write a Perl program that would use multiple cpu-cores?

Replies are listed 'Best First'.
Re: fork() on win10 - MCE
by Discipulus (Canon) on Jan 03, 2023 at 15:49 UTC
    Hello BernieC,

    > Is there some trick or something ...

    the trick is called MCE and its author lurks here around with a wagon of useful examples. MCE - Multi Core Engine is optimized to run on Windows too, all the pain is for the author :)


    There are no rules, there are no thumbs..
    Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
      Wow -- that module must've been a *bitch* to code up. But with Strawberry Perl on win10 I get:
      t/05_mce_child.t (Wstat: 768 Tests: 28 Failed: 3) Failed tests: 4-5, 8 Non-zero exit status: 3 Files=39, Tests=617, 9 wallclock secs ( 0.16 usr + 0.03 sys = 0.19 +CPU) Result: FAIL Failed 1/39 test programs. 3/617 subtests failed. gmake: *** [Makefile:816: test_dynamic] Error 255 Lockfile removed. MARIOROY/MCE-1.882.tar.gz C:\strawberry\c\bin\gmake.exe test -- NOT OK //hint// to see the cpan-testers results for installing this module, t +ry: reports MARIOROY/MCE-1.882.tar.gz Stopping: 'install' failed for 'MCE'. Failed during this command: MARIOROY/MCE-1.882.tar.gz : make_test NO
      I'm not sure what to do next

        Using Strawberry Perl v5.32.1 and cpanm I installed MCE on a Windows 11 machine and run the first sample from the synopsis for MCE without issue.

        C:\...>cpanm MCE --> Working on MCE Fetching + ... OK Configuring MCE-1.882 ... OK Building and testing MCE-1.882 ... OK Successfully installed MCE-1.882 1 distribution installed C:\...>perl -v This is perl 5, version 32, subversion 1 (v5.32.1) built for MSWin32-x +64-multi-thread

        Note that this is a 64 bit build of Perl (the x64 bit). What version of Perl are you using?

        Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond

        I wonder if sleep is broken or if the box is busy or delays coming from Windows anti-malware? Looking at tests 4, 5, and 8.

        perl -Ilib t/05_mce_child.t ... ok 2 - spawning asynchronously ok 3 - check for unique pids ok 4 - check list_running ok 5 - check list_joinable ok 6 - check list ok 7 - check pending ok 8 - check is_running child1 ok 9 - check is_joinable child1 ok 10 - check is_running child2 ok 11 - check is_joinable child2 ok 12 - check is_running child3 ok 13 - check is_joinable child3 ...

        Is there an alternative to sleep on the Windows platform, ensuring delay before the workers exit?

        { my ( $cnt, @list, %pids, %ret ); local $_; my ( $come_then_i_pray ) = ( "さあ、私&#123 +99;祈る" . "Ǣ" ); ok( 1, "spawning asynchronously" ); # test 2 MCE::Child->create( sub { sleep 1; sleep 1; "$come_then_i_pray $_" +} ) for ( 1..3 ); %pids = map { $_ => undef } MCE::Child->list_pids; is ( scalar( keys %pids ), 3, 'check for unique pids' ); # test 3 @list = MCE::Child->list_running; is ( scalar @list, 3, 'check list_running' ); # test 4 @list = MCE::Child->list_joinable; is ( scalar @list, 0, 'check list_joinable' ); # test 5 @list = MCE::Child->list; is ( scalar @list, 3, 'check list' ); # test 6 is ( MCE::Child->pending, 3, 'check pending' ); # test 7 $cnt = 0; for ( @list ) { ++$cnt; is ( $_->is_running, 1, 'check is_running child'.$cnt ); # te +sts 8, 10, 12 is ( $_->is_joinable, '', 'check is_joinable child'.$cnt ); # te +sts 9, 11, 13 } $cnt = 0; for ( @list ) { ++$cnt; $ret{ $_->join } = 1; is ( $_->error, undef, 'check error child'.$cnt ); } is ( scalar keys %ret, 3, 'check for unique values' ); for ( sort keys %ret ) { my $id = chop; s/ $//; is ( $_, $come_then_i_pray, "check for utf8 string $id" ); }; }

        Update 1: I have an idea to not use sleep. Instead, I will have workers read a channel. The manager process will notify workers to exit after completing tests list_running, list_joinable, list, pending, is_running, and is_joinable.

        Update 2: Released MCE 1.883 on meta::cpan

        Wild guess: Did you somehow get a version of strawberry that doesn't have multi-thread enabled?

        c:> perl -V:archname -V:myuname -V:osvers archname='MSWin32-x64-multi-thread'; myuname='Win32 strawberry-perl #1 Sun Jan 24 15:00:15 2021 x6 +4'; osvers='10.0.19042.746';

        or just the full perl -V to give the full information about your specific strawberry, in case perl -v and the three specific config variables above aren't sufficient to tell the other monks about your setup. (generally, if you ever think that there might be version or OS issues involved, sharing your full perl -V is best practice).

        and in your other post, you said, I don't exactly know what system "win32" is. Mine is a Win10/64bit.. that must make a difference.:

        Yes, win10/64bit could make a difference compared to 32bit or compared to win8/win7/.... But many of those passing results were win10/64bit: you can see that by clicking on the Win32 column header from the cpantesters report, which gives you a table of just those results, with more information. And each individual report will give perl -V or similar information, along with as much other meta-info as they can muster, which might help you determine the difference between your setup and theirs.

        I looked the reports and there were zillions, mostly all on Unix-based systems. there were a few PASSes on Win32 and no FAILs, I don't exactly know what system "win32" is. Mine is a Win10/64bit.. that must make a difference.
Re: fork() on win10
by Corion (Patriarch) on Jan 03, 2023 at 15:40 UTC

    In principle, fork() works on Windows, and should in principle use multiple threads (= CPUs).

    Depending on the actual work to be done or if there is resource contention, two instances of the program may still run slower, for example if both wait for the same spinning harddisk or try to access the same database or logfile.

    You can test if there is resource contention by simply starting your program twice in parallel and seeing whether it takes the same time or takes longer overall.

Re: fork() on win10
by karlgoethebier (Abbot) on Jan 04, 2023 at 18:10 UTC

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11149335]
Approved by Discipulus
Front-paged by kcott
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (3)
As of 2023-06-06 19:19 GMT
Find Nodes?
    Voting Booth?
    How often do you go to conferences?

    Results (29 votes). Check out past polls.