Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Improved reliability on the Windows platform with new MCE releases

by marioroy (Prior)
on Oct 10, 2022 at 09:02 UTC ( [id://11147319]=CUFP: print w/replies, xml ) Need Help??

MCE

1.880 Mon Oct 10 04:00:00 EST 2022 * Improved reliability on the Windows platform. * Improved MCE::Mutex::Channel::timedwait on the Windows platform. * Improved MCE::Mutex::Channel performance on UNIX platforms. * Resolved edge case in MCE::Child reaching deadlock.

MCE::Shared

1.878 Mon Oct 10 04:00:00 EST 2022 * Improved reliability on the Windows platform. * Added deeply-shared demonstration to POD.

Mutex

1.007 Mon Oct 10 04:00:00 EST 2022 * Improved reliability on the Windows platform. * Improved Mutex::Channel::timedwait on the Windows platform. * Improved Mutex::Channel performance on UNIX platforms.

Replies are listed 'Best First'.
Re: Improved reliability on the Windows platform with new MCE releases
by marioroy (Prior) on Oct 14, 2022 at 06:52 UTC

    MCE 1.881 update

    I read code recently, using MCE::Map with 4 workers. But, only 2 workers were running. The update is a tiny edit to the _parse_chunk_size function.

    1.881 Thu Oct 13 23:45:00 EST 2022 * Improved the private _parse_chunk_size function. For better utilization of CPU cores in MCE::Grep, MCE::Map, and MCE::Stream, processing small input sizes. Previously, chunk_size => 'auto' equals 2 minimally. Starting with MCE v1.881, 'auto' equals 1 minimally.

    It has been a while and forgotten what the overhead is running mce_map. Testing was done on a Linux box. Also, Perl has Sereal::Encoder and Sereal::Decoder installed.

    native map

    use strict; use warnings; use Time::HiRes 'time'; my $start = time(); my @ret = map { $_ * 2 } 1..10_000_000; printf "%s %s\n", $ret[5_000_000], $ret[-1]; printf "%0.3f seconds\n", time() - $start; __END__ 10000002 20000000 0.561 seconds

    mce_map

    Workers request the next chunk from the manager process. The overhead is low, considering mce_map involves bidirectional data transfers (from manager, to manager).

    use strict; use warnings; use MCE::Map; use Time::HiRes 'time'; MCE::Map->init( chunk_size => 'auto', max_workers => 8, posix_exit => 1, ); my $start = time(); my @ret = mce_map { $_ * 2 } 1..10_000_000; printf "%s %s\n", $ret[5_000_000], $ret[-1]; printf "%0.3f seconds\n", time() - $start; __END__ 10000002 20000000 0.996 seconds

    mce_map_s

    For a sequence of numbers, append the suffix "_s" to "mce_map". Workers do not involve the manager process for receiving input. Instead, workers communicate the next starting number among themselves. The syntax is slightly different. Specify the starting and ending number after the code block (two args, notice the comma). Data transfer is unidirectional (to manager).

    use strict; use warnings; use MCE::Map; use Time::HiRes 'time'; MCE::Map->init( chunk_size => 'auto', max_workers => 8, posix_exit => 1, ); my $start = time(); my @ret = mce_map_s { $_ * 2 } 1, 10_000_000; printf "%s %s\n", $ret[5_000_000], $ret[-1]; printf "%0.3f seconds\n", time() - $start; __END__ 10000002 20000000 0.655 seconds

    something different

    I have seen the following MCE usage. Here, chunk_size is explicitly set to 1 behind the scene. One may omit the first two lines inside the code block.

    use strict; use warnings; use Data::Dumper; use MCE; use Time::HiRes 'sleep'; my %res = (); my $mce = MCE->new( max_workers => 5, posix_exit => 1, gather => \%res, ); my @input = 1..10; $mce->foreach(\@input, sub { # my ($mce, $chunk_ref, $chunk_id) = @_; # my $item = $chunk_ref->[0]; # or $_ my $item = $_; MCE->printf("Worker %s began item $item\n", MCE->wid); sleep 0.01; # simulate work MCE->gather($item => $item * 2); MCE->printf("Worker %s ended item $item\n", MCE->wid); }); $mce->shutdown; print Dumper(\%res);

    output

    Worker 5 began item 3 Worker 4 began item 2 Worker 1 began item 1 Worker 2 began item 4 Worker 3 began item 5 Worker 5 ended item 3 Worker 4 ended item 2 Worker 1 ended item 1 Worker 5 began item 6 Worker 3 ended item 5 Worker 2 ended item 4 Worker 1 began item 8 Worker 4 began item 7 Worker 2 began item 9 Worker 3 began item 10 Worker 5 ended item 6 Worker 1 ended item 8 Worker 4 ended item 7 Worker 3 ended item 10 Worker 2 ended item 9 $VAR1 = { '9' => 18, '7' => 14, '3' => 6, '1' => 2, '5' => 10, '8' => 16, '4' => 8, '2' => 4, '10' => 20, '6' => 12 };

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: CUFP [id://11147319]
Approved by Corion
Front-paged by Discipulus
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others imbibing at the Monastery: (5)
As of 2024-03-28 16:23 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found