|
Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:
Monks,
I am trying to get the result of a shell command while using threads but what else is new - I am not having any luck. Some of my commands take a long time so I like to stop them and report the event. Is there a better way to do this using another technique?
use Thread qw(yield async);
async {
my $time = 10;
while ($time--){$command = `shell command`;}
yield;
};
#do something with command
Thank you. |
Re: Threads or no Threads (total timeout)
by ikegami (Patriarch) on Feb 24, 2007 at 06:43 UTC
|
use strict;
use warnings;
use IO::Select qw( );
use IPC::Open2 qw( open2 );
use Time::HiRes qw( time ); # Optional
my $timeout = 30.000; # Abort after this many seconds.
my $command = '...'; # The command to execute.
my $from_child = ''; # Receives the output of the command.
my $pid = open2(undef, my $fh_from_child, $command);
my $sel = IO::Select->new($fh_from_child);
my $abort_time = time + $timeout;
for (;;) {
my $time_left = $abort_time - time;
my @r = $time_left > 0 ? $sel->can_read($time_left): ();
if (!@r) {
kill(TERM => $pid);
die("Child took more than $timeout seconds.\n");
}
read($fh_from_child, $from_child, 4096, length($from_child))
or last;
}
print($from_child);
waitpid($pid, 0);
Also tested. Also won't work on Windows.
Update: Added call to kill. Could still use some work there.
Update: Or maybe you want to give a notice at a fixed interval? It should be easy to combine the techniques used in each of my posts to do this.
| [reply]
[d/l]
[select] |
Re: Threads or no Threads (idle timeout)
by ikegami (Patriarch) on Feb 24, 2007 at 06:27 UTC
|
I'm not sure what you want. Are you looking to be notifie when the child has been idle for a certain amount of time?
use strict;
use warnings;
use IO::Select qw( );
my $timeout = 30.000; # Notify user after idle for this many secs
my $command = '...'; # The command to execute.
my $from_child = ''; # Receives the output of the command.
open(my $fh_from_child, "$command |")
or die("Unable to launch $command: $!\n");
my $sel = IO::Select->new($fh_from_child);
for (;;) {
my @r = $sel->can_read($timeout);
if (!@r) {
print("Another $timeout seconds has gone by...\n");
redo;
}
read($fh_from_child, $from_child, 4096, length($from_child))
or last;
}
print($from_child);
Tested on FreeBSD. It won't work on Windows (since you can't select pipes).
| [reply]
[d/l]
[select] |
Re: Threads or no Threads
by BrowserUk (Patriarch) on Feb 24, 2007 at 09:42 UTC
|
#! perl -slw
use strict;
our $WAIT ||= 10;
my $cmd = 'perl -e"$|++;sleep 1, print for 1 .. 10" |';
my $pid = open my $cmdFh, $cmd or die "Couldn't run '$cmd': $!";
sleep 1 while kill 0, $pid and $WAIT--;
kill 3, $pid and warn 'Command timed out' if $WAIT;
my $results = do{ local $/; <$cmdFh> };
print "Got: '$results'";
A few simple tests
c:\test>junk9 -WAIT=2
Command timed out at c:\test\junk9.pl line 11.
Got: '12'
c:\test>junk9 -WAIT=9
Command timed out at c:\test\junk9.pl line 11.
Got: '12345678'
c:\test>junk9 -WAIT=10
Got: '12345678910'
c:\test>junk9 -WAIT=11
Got: '12345678910'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply]
[d/l]
[select] |
Re: Threads or no Threads (Threading, Forking, Signals, Event Loop and Concurrency References)
by eyepopslikeamosquito (Archbishop) on Feb 24, 2007 at 09:12 UTC
|
In case it's of use, I wrote a couple of scripts to run a command, killing it if it took more than a specified timeout, and reporting how long it took to run:
Update: Using this node nowadays for links on Threading and Concurrency.
Perl Documentation
CPAN
Perl monk marioroy has produced some excellent CPAN modules in this domain:
- MCE by marioroy (Many-Core Engine for Perl providing parallel processing capabilities - can be configured to use processes or threads (via use_threads parameter), by default MCE spawns child processes on UNIX platforms and threads on Windows)
- MCE::Grep by marioroy
- Mutex by marioroy
- MCE::Shared by marioroy
See also:
Event-driven Programming
Event-driven programming is an attractive alternative to threads, especially when using scripting languages.
For Perl, Mojolicious seems to be the most popular nowadays, as described at:
See also:
Chunking
As noted by marioroy, chunking complements event-driven programming.
Some further chunking examples from marioroy:
See also: event-driven references in the Wikipedia References section below.
TLS/SSL Related
On CPAN:
Some Perl Monks Threading and Concurrency Nodes
- IPC::Open, Parallel::ForkManager, or Sockets::IO for parallelizing? by mldvx4 (Sep 2023) - replies mention Mojolicious, Mojo::Promise, Net::Async::HTTP and Future (by Paul Evans), LWP::Parallel, MCE, Thread::Queue, Forking::Amazing, ...
- Perl and Threading by scorpion7 (Nov 2023) - panic: del_backref, *svp=0 phase=DESTRUCT refcnt=3 during global destruction; choroba reply recommends Thread::Queue
Windows and Strawberry Perl
- Strawberry Perl (wikipedia)
- Mingw-w64 (wikipedia) - gcc-based C/C++ compiler used by Strawberry Perl (Note: MSYS2 ("minimal system 2") is a software distribution and a development platform for Microsoft Windows, based on Mingw-w64 and Cygwin ... MSYS2 and its bash environment is used by Git)
- Cygwin (wikipedia) - a Unix-like environment and command-line interface for Microsoft Windows (uses cross-compilers from Mingw-w64 project)
- How is fork() implemented in Cygwin? - Cygwin fork() essentially works like a non-copy on write version of fork() (like old Unix versions used to do). Because of this it can be a little slow. In most cases, you are better off using the spawn family of calls if possible.
- Strawberry Perl (official site)
- Strawberry Perl (github)
Parallel::ForkManager
Promises and Futures
- Future by Paul Evans
- Promises by Yanick Champoux (compares to Scala Promises and Futures)
- C++ std::promise
- C++ std::future - future and promise enable transfer of a value between two tasks without explicit use of a lock (the "system" implements the transfer efficiently); when a task wants to pass a value to another, it puts the value into a promise; the system makes that value appear in the corresponding future; promise provides simple put operations to match future's get
Signals
C++ Multi-Threading
- NVIDIA HPC SDK Version 24.3 Documentation (nvc++ is a C++17 compiler for NVIDIA GPUs and AMD, Intel, OpenPOWER, and Arm CPUs. It invokes the C++ compiler, assembler, and linker for the target processors with options derived from its command line arguments. nvc++ supports ISO C++17, supports GPU and multicore CPU programming with C++17 parallel algorithms, OpenACC, and OpenMP)
- The three compilers used by marioroy in Long List is Long are g++, clang++, nvc++. I also used Google abseil library.
Unit Testing Concurrent Code
Test-driven development, a practice enabling developers
to detect bugs early by incorporating unit testing into the development process, has become
wide-spread, but it has only been effective for programs with a single thread of control.
The order of operations in different threads is essentially non-deterministic, making it
more complicated to reason about program properties in concurrent programs than in
single-threaded programs.
-- from a recent PhD proposal to develop a concurrent testing framework
See the "Testing Concurrent Software References" section below for more references
in this active area of research.
Though I haven't used any of these tools yet, I'd be interested to hear from folks who have
or who have general advice and tips on how to troubleshoot and fix complex concurrency-related bugs.
In particular, I'm not aware of any Perl-based concurrent testing frameworks.
In practice, the most effective, if crude, method I've found for dealing
with nasty concurrency bugs is good tracing code at just the right places combined
with understanding and reasoning about the code, performing experiments,
and "thinking like a detective".
One especially useful experiment (mentioned in
Clean Code)
is to add "jiggle points" at critical places in your concurrent code and have
the jiggle point either do nothing, yield, or sleep for a short interval.
There are more sophisticated tools available, for example IBM's ConTest,
that use this approach to flush out bugs in concurrent code.
Testing Concurrent Software References
Wikipedia References
See Also
Updated: Extra references added long after the original reply was made.
| [reply]
[d/l]
[select] |
Re: Threads or no Threads
by bsdz (Friar) on Feb 24, 2007 at 13:19 UTC
|
If you are using Windows, you may want to consider using Win32::Job. | [reply] |
Re: Threads or no Threads
by Anonymous Monk on Feb 24, 2007 at 16:55 UTC
|
Thank you guys for your replies. I am actually testing this on Windows but will be using it on Linux. This thread has been very educational. I appreciate it. | [reply] |
|
|
