Re: script hanging at thread start [MCE::Flow alternative]
by 1nickt (Canon) on Nov 24, 2020 at 23:05 UTC
|
use strict;
use warnings;
use MCE::Flow chunk_size => 1;
mce_flow sub {
chomp;
printf("%s processing %s\n", $$, $_);
if (/(\w.+\w\d+.+net),(\w+)/) {
my $host = $1;
my $type = $2;
printf "%s connecting to %s\n", $$, $host;
}
printf "%s finished %s\n", $$, $_;
}, [(<DATA>)];
__DATA__
server1.net,value
server2.net,value
server3.net,value
server4.net,value
Output:
perl 11124156.pl
597 processing server3.net,value
605 processing server2.net,value
595 processing server1.net,value
600 processing server4.net,value
605 connecting to server2.net
595 connecting to server1.net
600 connecting to server4.net
605 finished server2.net,value
595 finished server1.net,value
600 finished server4.net,value
597 connecting to server3.net
597 finished server3.net,value
Hope this helps!
The way forward always starts with a minimal test.
| [reply] [Watch: Dir/Any] [d/l] [select] |
|
Thanks! It will take me some time to learn MCE, but that looks promising.
| [reply] [Watch: Dir/Any] |
|
Kinda code reuse
«The Crux of the Biscuit is the Apostrophe»
perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help
| [reply] [Watch: Dir/Any] [d/l] |
|
how does it not use threads?
| [reply] [Watch: Dir/Any] |
Re: script hanging at thread start
by GrandFather (Saint) on Nov 24, 2020 at 20:39 UTC
|
use warnings;
use strict;
use threads;
my @Threads;
for my $Serv (<DATA>) {
chomp $Serv;
print "Created thread for $Serv\n";
push @Threads, threads->new(sub {runMain($Serv)});
}
$_->join() for @Threads;
print "Work complete\n";
sub runMain {
my ($server) = @_;
print "Starting runMain('$server')\n";
if ($server =~ /(\w.+\w\d+.+net),(\w+)/) {
my $Server = $1;
my $Type = $2;
print "Connecting to $Server\n";
}
print "Done for $server\n";
}
__DATA__
server1.net,value
server2.net,value
server3.net,value
server4.net,value
it prints:
Created thread for server1.net,value
Created thread for server2.net,value
Starting runMain('server1.net,value')
Connecting to server1.net
Done for server1.net,value
Created thread for server3.net,value
Starting runMain('server2.net,value')
Connecting to server2.net
Done for server2.net,value
Created thread for server4.net,value
Starting runMain('server3.net,value')
Connecting to server3.net
Done for server3.net,value
Starting runMain('server4.net,value')
Connecting to server4.net
Done for server4.net,value
Work complete
How is your code different?
Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
| [reply] [Watch: Dir/Any] [d/l] [select] |
|
In your example I get Created thread for server1 - 4, but only 3 will get the other output. For example I would not see Starting runMain('server3.net,value'), Connecting to server3.net, Done for server3.net.
| [reply] [Watch: Dir/Any] |
|
| [reply] [Watch: Dir/Any] |
Re: script hanging at thread start
by ikegami (Patriarch) on Nov 27, 2020 at 00:27 UTC
|
LWP is really slow. If you were planning on doing HTTP requests with LWP, you might be able to solve both problems at once by using Net::Curl::Multi
| [reply] [Watch: Dir/Any] |
Re: script hanging at thread start
by karlgoethebier (Abbot) on Nov 25, 2020 at 12:22 UTC
|
Probably this is also of interest: If you use something from the MCE family as 1nickt suggested (a real good idea) you can decide if your app uses threads or forks if you say use threads or not before you call something from MCE. As far as I remember. Please see Re: Parallel trigger of different subs. You may try it and watch your process table. Best regards, Karl
«The Crux of the Biscuit is the Apostrophe»
perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help
| [reply] [Watch: Dir/Any] [d/l] [select] |
Re: script hanging at thread start
by Anonymous Monk on Dec 03, 2020 at 00:41 UTC
|
#!/usr/bin/perl --
use warnings;
use strict;
use threads;
use threads stack_size => 4096, 'exit' => 'threads_only';;
Main( @ARGV );
exit( 0 );
sub Main {
open my( $data ), '<', \'not,match
server1.net,value
server2.net,value
server3.net,value
server4.net,value
';
while( my $Serv = <$data> ) {
chomp $Serv;
my $thread = threads->new( \&runMoo, $Serv );
print tidt( $thread ), "Created thread for $Serv \n";
}
while( threads->list ) {
for my $thread ( threads->list( threads::joinable ) ) {
print tidt( $thread ), "Joining ", $thread->join, "\n";
}
Win32::Sleep( 33 );
}
print tidt(), "Work complete\n";
}
sub runMoo {
my( $server ) = @_;
Win32::Sleep( rand 4000 );
print tidt(), "Starting runMoo('$server')\n";
if( $server =~ /(\w.+\w\d+.+net),(\w+)/ ) {
my $Server = $1;
my $Type = $2;
print tidt(), "Connecting to $Server\n";
Win32::Sleep( rand 2000 );
} else {
print tidt(), "not match server ($server)\n";
}
print tidt(), "Done for $server\n";
}
sub tidt {
my( $thread ) = ( @_, threads->self );
my $tid = $thread->tid;
return join '', $tid, " " x $tid, $tid, " ";
}
__END__
1 1 Created thread for not,match
2 2 Created thread for server1.net,value
3 3 Created thread for server2.net,value
4 4 Created thread for server3.net,value
5 5 Created thread for server4.net,value
5 5 Starting runMoo('server4.net,value')
5 5 Connecting to server4.net
3 3 Starting runMoo('server2.net,value')
3 3 Connecting to server2.net
1 1 Starting runMoo('not,match')
1 1 not match server (not,match)
1 1 Done for not,match
1 1 Joining 1
5 5 Done for server4.net,value
5 5 Joining 1
3 3 Done for server2.net,value
3 3 Joining 1
4 4 Starting runMoo('server3.net,value')
4 4 Connecting to server3.net
2 2 Starting runMoo('server1.net,value')
2 2 Connecting to server1.net
4 4 Done for server3.net,value
4 4 Joining 1
2 2 Done for server1.net,value
2 2 Joining 1
00 Work complete
| [reply] [Watch: Dir/Any] [d/l] |
Re: script hanging at thread start
by perlfan (Vicar) on Nov 30, 2020 at 19:08 UTC
|
Without seeing the rest of your code, you would do well to throttle the number of ithreads (not real "threads") being active at any given time. And since you're dealing with external services, I would also set an alarm (timeout) inside of your anonymous sub that you're using to call runMain. | [reply] [Watch: Dir/Any] [d/l] |
|
| [reply] [Watch: Dir/Any] |
Re: script hanging at thread start
by Anonymous Monk on Dec 02, 2020 at 10:24 UTC
|
| [reply] [Watch: Dir/Any] |
Re: script hanging at thread start
by Anonymous Monk on Nov 25, 2020 at 15:32 UTC
|
From perldoc threads
WARNING
The "interpreter-based threads" provided by Perl are not the fast, lightweight system for multitasking that one might expect or hope for. Threads are implemented in a way that makes them easy to misuse. Few people know how to use them correctly or will be able to provide help.
The use of interpreter-based threads in perl is officially discouraged.
| [reply] [Watch: Dir/Any] [d/l] |
|
| [reply] [Watch: Dir/Any] |
|
| [reply] [Watch: Dir/Any] [d/l] |
|
I know very little about threads but it strikes me that this problem is better suited to fork
(assuming this is on a *IX OS)
edit
I've just reread the original question and it's Strawberry Perl which I believe implements fork with threads
| [reply] [Watch: Dir/Any] [d/l] [select] |