Re: Multi-thread combining the results together

Inbound two MCE examples in response to monks Discipulus and 1nickt suggesting MCE. The 1st example stores to a hash.

use strict;
use warnings;

use MCE::Loop;
use Data::Dumper;

my @tokens = qw( ab bc cd de ef fg gh );

sub build_regex {
    my ($token) = @_;
    chop $token;
}

MCE::Loop->init(
    chunk_size  => 1,
    max_workers => 4,
);

my %result = mce_loop {
    my $token = $_;
    my $regex = build_regex($token);
    my @line_results = grep { $_ ne $token and /$regex/ } @tokens;
    MCE->gather( $token => \@line_results ) if @line_results;
} @tokens;

MCE::Loop->finish;

print Dumper(\%result), "\n";

__END__

$VAR1 = {
          'ef' => [
                    'fg'
                  ],
          'fg' => [
                    'gh'
                  ],
          'bc' => [
                    'cd'
                  ],
          'ab' => [
                    'bc'
                  ],
          'cd' => [
                    'de'
                  ],
          'de' => [
                    'ef'
                  ]
        };
[download]

The 2nd example stores to an array and maintains order, possible with MCE::Relay which runs orderly.

use strict;
use warnings;

use MCE::Loop;
use Data::Dumper;

my @tokens = qw( ab bc cd de ef fg gh );

sub build_regex {
    my ($token) = @_;
    chop $token;
}

MCE::Loop->init(
    chunk_size  => 1,
    max_workers => 4,
    init_relay  => 1,
);

my @result = mce_loop {
    my $token = $_;
    my $regex = build_regex($token);
    my @line_results = grep { $_ ne $token and /$regex/ } @tokens;
    MCE::relay {
        MCE->gather(\@line_results) if @line_results;
    };
} @tokens;

MCE::Loop->finish;

print Dumper(\@result), "\n";

__END__

$VAR1 = [
          [
            'bc'
          ],
          [
            'cd'
          ],
          [
            'de'
          ],
          [
            'ef'
          ],
          [
            'fg'
          ],
          [
            'gh'
          ]
        ];
[download]

Regards, Mario

Comment on Re: Multi-thread combining the results together Select or Download Code

Replies are listed 'Best First'.

Re^2: Multi-thread combining the results together
by marioroy (Prior) on Jul 25, 2019 at 12:18 UTC

Sharing data is feasible using a parallel module of your choice. Here is a demonstration using Parallel::ForkManager and MCE::Shared.

use strict;
use warnings;

use Parallel::ForkManager;
use MCE::Shared;
use Data::Dumper;

my @tokens = qw( ab bc cd de ef fg gh );

sub build_regex {
    my ($token) = @_;
    chop $token;
}

my $pm = Parallel::ForkManager->new(4);
$pm->set_waitpid_blocking_sleep(0);

tie my %result, 'MCE::Shared';

DATA_LOOP:
foreach my $token ( @tokens ) {
    # forks and returns the pid for the child
    my $pid = $pm->start and next DATA_LOOP;

    my $regex = build_regex($token);
    my @line_results = grep { $_ ne $token and /$regex/ } @tokens;
    $result{ $token } = \@line_results if @line_results;

    $pm->finish(0);
}

$pm->wait_all_children;

print Dumper( tied(%result)->export({ unbless => 1 }) ), "\n";

__END__

$VAR1 = {
          'ef' => [
                    'fg'
                  ],
          'fg' => [
                    'gh'
                  ],
          'cd' => [
                    'de'
                  ],
          'de' => [
                    'ef'
                  ],
          'bc' => [
                    'cd'
                  ],
          'ab' => [
                    'bc'
                  ]
        };
[download]

Regards, Mario

[reply]
[d/l]


Perl Monk, Perl Meditation
	PerlMonks