http://qs321.pair.com?node_id=11124156

armcinto has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,
I'm stuck and need some direction. I have a new script that is frequently hanging and needs to be killed. The script reads in a file with a list of hostnames (4-5) and creates a thread for each hostname. I can see the print statement that each thread is created, but I don't see the first print statement in the runMain subroutine. Most of threads complete the entire script, but 1-2 will appear to never start. Not sure what else to try at this point to get more information. This is Strawberry Perl (64-bit) 5.28.1.1-64bit environment.

sub runMain { print "Starting runMain on $_[0]\n"; if ($_[0] =~ /(\w.+\w\d+.+net),(\w+)/) { $Server = $1; $Type = $2; print "Connecting to $Server\n"; } } open (LIST, "list.txt"); for my $Serv (<LIST>) { print "Created thread for $Serv\n"; push @Threads, threads->new(sub {runMain($Serv)}); }


Thanks,
-A

Replies are listed 'Best First'.
Re: script hanging at thread start [MCE::Flow alternative]
by 1nickt (Canon) on Nov 24, 2020 at 23:05 UTC

    Hi, here's something that does not require threads, powered by the Perl Many-Core Engine:

    use strict; use warnings; use MCE::Flow chunk_size => 1; mce_flow sub { chomp; printf("%s processing %s\n", $$, $_); if (/(\w.+\w\d+.+net),(\w+)/) { my $host = $1; my $type = $2; printf "%s connecting to %s\n", $$, $host; } printf "%s finished %s\n", $$, $_; }, [(<DATA>)]; __DATA__ server1.net,value server2.net,value server3.net,value server4.net,value
    Output:
    perl 11124156.pl 597 processing server3.net,value 605 processing server2.net,value 595 processing server1.net,value 600 processing server4.net,value 605 connecting to server2.net 595 connecting to server1.net 600 connecting to server4.net 605 finished server2.net,value 595 finished server1.net,value 600 finished server4.net,value 597 connecting to server3.net 597 finished server3.net,value

    Hope this helps!


    The way forward always starts with a minimal test.
      Thanks! It will take me some time to learn MCE, but that looks promising.

        Kinda code reuse

        «The Crux of the Biscuit is the Apostrophe»

        perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

      how does it not use threads?
Re: script hanging at thread start
by GrandFather (Saint) on Nov 24, 2020 at 20:39 UTC

    When I run:

    use warnings; use strict; use threads; my @Threads; for my $Serv (<DATA>) { chomp $Serv; print "Created thread for $Serv\n"; push @Threads, threads->new(sub {runMain($Serv)}); } $_->join() for @Threads; print "Work complete\n"; sub runMain { my ($server) = @_; print "Starting runMain('$server')\n"; if ($server =~ /(\w.+\w\d+.+net),(\w+)/) { my $Server = $1; my $Type = $2; print "Connecting to $Server\n"; } print "Done for $server\n"; } __DATA__ server1.net,value server2.net,value server3.net,value server4.net,value

    it prints:

    Created thread for server1.net,value Created thread for server2.net,value Starting runMain('server1.net,value') Connecting to server1.net Done for server1.net,value Created thread for server3.net,value Starting runMain('server2.net,value') Connecting to server2.net Done for server2.net,value Created thread for server4.net,value Starting runMain('server3.net,value') Connecting to server3.net Done for server3.net,value Starting runMain('server4.net,value') Connecting to server4.net Done for server4.net,value Work complete

    How is your code different?

    Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
      In your example I get Created thread for server1 - 4, but only 3 will get the other output. For example I would not see Starting runMain('server3.net,value'), Connecting to server3.net, Done for server3.net.

        A bug in threads maybe? I'm running "This is perl 5, version 32, subversion 0 (v5.32.0) built for MSWin32-x64-multi-thread".

        Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
Re: script hanging at thread start
by ikegami (Patriarch) on Nov 27, 2020 at 00:27 UTC

    LWP is really slow. If you were planning on doing HTTP requests with LWP, you might be able to solve both problems at once by using Net::Curl::Multi

Re: script hanging at thread start
by karlgoethebier (Abbot) on Nov 25, 2020 at 12:22 UTC

    Probably this is also of interest: If you use something from the MCE family as 1nickt suggested (a real good idea) you can decide if your app uses threads or forks if you say use threads or not before you call something from MCE. As far as I remember. Please see Re: Parallel trigger of different subs. You may try it and watch your process table. Best regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

    perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

Re: script hanging at thread start
by Anonymous Monk on Dec 03, 2020 at 00:41 UTC

    Hi,

    Is each thread actually created? What do you get with this wordy program

    #!/usr/bin/perl -- use warnings; use strict; use threads; use threads stack_size => 4096, 'exit' => 'threads_only';; Main( @ARGV ); exit( 0 ); sub Main { open my( $data ), '<', \'not,match server1.net,value server2.net,value server3.net,value server4.net,value '; while( my $Serv = <$data> ) { chomp $Serv; my $thread = threads->new( \&runMoo, $Serv ); print tidt( $thread ), "Created thread for $Serv \n"; } while( threads->list ) { for my $thread ( threads->list( threads::joinable ) ) { print tidt( $thread ), "Joining ", $thread->join, "\n"; } Win32::Sleep( 33 ); } print tidt(), "Work complete\n"; } sub runMoo { my( $server ) = @_; Win32::Sleep( rand 4000 ); print tidt(), "Starting runMoo('$server')\n"; if( $server =~ /(\w.+\w\d+.+net),(\w+)/ ) { my $Server = $1; my $Type = $2; print tidt(), "Connecting to $Server\n"; Win32::Sleep( rand 2000 ); } else { print tidt(), "not match server ($server)\n"; } print tidt(), "Done for $server\n"; } sub tidt { my( $thread ) = ( @_, threads->self ); my $tid = $thread->tid; return join '', $tid, " " x $tid, $tid, " "; } __END__ 1 1 Created thread for not,match 2 2 Created thread for server1.net,value 3 3 Created thread for server2.net,value 4 4 Created thread for server3.net,value 5 5 Created thread for server4.net,value 5 5 Starting runMoo('server4.net,value') 5 5 Connecting to server4.net 3 3 Starting runMoo('server2.net,value') 3 3 Connecting to server2.net 1 1 Starting runMoo('not,match') 1 1 not match server (not,match) 1 1 Done for not,match 1 1 Joining 1 5 5 Done for server4.net,value 5 5 Joining 1 3 3 Done for server2.net,value 3 3 Joining 1 4 4 Starting runMoo('server3.net,value') 4 4 Connecting to server3.net 2 2 Starting runMoo('server1.net,value') 2 2 Connecting to server1.net 4 4 Done for server3.net,value 4 4 Joining 1 2 2 Done for server1.net,value 2 2 Joining 1 00 Work complete
Re: script hanging at thread start
by perlfan (Vicar) on Nov 30, 2020 at 19:08 UTC
    Without seeing the rest of your code, you would do well to throttle the number of ithreads (not real "threads") being active at any given time. And since you're dealing with external services, I would also set an alarm (timeout) inside of your anonymous sub that you're using to call runMain.
      Alarm has no place
Re: script hanging at thread start
by Anonymous Monk on Dec 02, 2020 at 10:24 UTC
Re: script hanging at thread start
by Anonymous Monk on Nov 25, 2020 at 15:32 UTC

    From perldoc threads

    WARNING

    The "interpreter-based threads" provided by Perl are not the fast, lightweight system for multitasking that one might expect or hope for. Threads are implemented in a way that makes them easy to misuse. Few people know how to use them correctly or will be able to provide help.

    The use of interpreter-based threads in perl is officially discouraged.

        Ouch! "It" is still there. Big surprise.

        «The Crux of the Biscuit is the Apostrophe»

        perl -MCrypt::CBC -E 'say Crypt::CBC->new(-key=>'kgb',-cipher=>"Blowfish")->decrypt_hex($ENV{KARL});'Help

      I know very little about threads but it strikes me that this problem is better suited to fork
      (assuming this is on a *IX OS)

      edit

      I've just reread the original question and it's Strawberry Perl which I believe implements fork with threads