mmoorreett has asked for the wisdom of the Perl Monks concerning the following question:
Hi All
I have been developing relatively simple perl programs on and off over the years, mainly for integrations using synchronous operations. I have since been tasked with building a "simple" ICMP and SNMP polling engine (yes re-inventing the wheel...) to allow for asynchronous/parallel polling. In researching the various options, i have learnt of the event loop options in perl. AnyEvent, IO:Async etc. Given AnyEvent has Ping and SNMP modules I have started using these and learning the ropes of AnyEvent. In testing the use cases required for the solution, I need to be able to ping different sets of hosts on different timers. So I have started developing the below code. The problem I am having is i'm getting the "AnyEvent::CondVar: recursive blocking wait attempted" error. I understand why, $cv_ping->recv; is blocking the event loop. However I am not how I work around this, or if my approach is actually going to work as I don't have much experience with AnyEvent. I have trawled the interwebs for various materials and tried different things but haven't been able to get it to work.
My code is below, help/advice is much appreciated.
use strict;
use AnyEvent;
use AnyEvent::Ping;
use IO::Async::Timer::Periodic;
use IO::Async::Loop;
use Data::Dumper;
my @todoList = map { "192.168.100." . $_ } (1 .. 254);
my $ping_timeout=2; #seconds to wait for response
my $ping_attemps=4; #number of pings to issue,
my $ping_pktsize=32; #packet size of ping
my $start_run = time();
main();
sub main{
my $loop = IO::Async::Loop->new;
my $timer = IO::Async::Timer::Periodic->new(
interval => 5,
first_interval => 2,
reschedule => 'hard',
on_tick => sub {
print "before ping\n";
\&ping_hosts(@todoList);
print "after ping\n";
},
);
my $timer2 = IO::Async::Timer::Periodic->new(
interval => 7,
first_interval => 2,
reschedule => 'hard',
on_tick => sub {
print "before ping2\n";
\&ping_hosts(@todoList);
print "after ping2\n";
},
);
$timer->start;
$timer2->start;
$loop->add( $timer );
$loop->add( $timer2 );
$loop->run;
}
sub ping_hosts{
my @hosts=@_;
my $callback=shift;
my $result;
my $cv_ping = AnyEvent->condvar;
my $ping = AnyEvent::Ping->new(packet_size => $ping_pktsize);
$ping->timeout($ping_timeout);
#$cv_ping->begin(sub { shift->send($_[0]) });
foreach my $host (@hosts) {
$cv_ping->begin;
my $request;
$request = $ping->ping($host, $ping_attemps, sub {
print "\n$host state:".Dumper($_[0]);
undef $request;
$cv_ping->end;
});
}
$cv_ping->recv;
$cv_ping->end;
#$cv_ping->cb(sub { $_[0]->recv });
$ping->end;
}
Re: Implementing AnyEvent::Ping multiple times
by hippo (Bishop) on Jun 11, 2020 at 11:43 UTC
|
The code as presented is confusing to me. That may be my fault as I am no authority on event loops. However, since nobody else has replied yet it may be that they are confused as well. If you could maybe address these questions I have then it might encourage others to chip in too.
- You are using both IO::Async and AnyEvent but not using either of the glue modules I would expect to see (AnyEvent::Impl::IOAsync
or IO::Async::Loop::AnyEvent). Could you explain this?
- The code appears to want to ping the same set of hosts twice on different schedules. I cannot think of any reason to do this. What is your intention here?
- Within each timer you are pinging 254 addresses 4 times with a 2 second timeout on each. Any target which drops the packets will therefore take 8 seconds to complete. And yet your timers are each on a hard reschedule of less than 8 seconds. Even if your code worked without blocking you could end up with a ping bomb. Again, without knowing your intention it is hard to advise further.
If you could give a clear statement of your intended algorithm it would help me (and maybe others) to be in a position to help you more.
Update: removed reference to IO::Async::Loop::AnyEvent as newer versions of AnyEvent explicitly prevent its use.
| [reply] |
|
Thanks for the reply, sorry if this seems confusing.
This is my first go at using event loops so I am slowly grasping the concepts and how to implement it, but i'm struggling a little, so I'm not even sure if i'm doing this properly.
The main reason I started with this, was to develop a simple perl ICMP monitoring plugin for another application that would be able to ping different sets of many hosts, very quickly, across different, possibly overlapping schedules. E.g 192.168.1.x every 2 minutes and 192.168.2.x every 5 minutes. I've tried various methods and the AnyEvent::Ping module seems to be the best option for performance for asynchronous. To answer your questions:
1. I used IO::Async::Timer::Periodic to give me an extra option on scheduling the code execution. I didn't realise that i needed AnyEvent::Impl::IOAsync, i've done some more reading and understand this now, so thanks.
2. This is just test code, so in reality the intention is to invoke AnyEvent::Ping on different sets of hosts and different schedules, however I'm not sure how I get around the fact that AnyEvent::Ping is blocking. I just didn't bother to create another subnet as I'm still trying to understand how I will get this to work. So just ignore that for now.
3. Yes I'm aware of that, thats me testing on how the reschedule option in Periodic works, i.e. the ping bomb you're talking about, so just ignore that for now.
If this is not the best approach to achieve this then I'm open to alternatives. I also looked at using Threads but its very CPU/memory intensive compared to AnyEvent::Ping. I'm testing this on Windows using Strawberry Perl if thats relevant. Thanks!
Mike
| [reply] |
|
Thanks for the clarifications - that helps a lot. Having played around with this now I think that the problem you are having essentially boils down to the fact that the event loop is blocking because you are telling it to and that there's no reason for you to do so.
Firstly, in order to simplify things I have decided to concentrate solely on AnyEvent (because that's what I'm less unfamiliar with) and ditch IO::Async for this illustration. That gets rid of the need for the glue and stops the chances of 2 different event loops coming to blows. As a starting point, here are two timers running on 5 and 7 second intervals implemented purely with AnyEvent. There's a third timer there just so the thing doesn't carry on forever.
#!/usr/bin/env perl
use strict;
use warnings;
use AnyEvent;
my $cv = AnyEvent->condvar;
my @timers = (
AnyEvent->timer (
after => 2,
interval => 5,
cb => sub {
warn "Timer 1 at " . AnyEvent->now . "\n";
}
),
AnyEvent->timer (
after => 2,
interval => 7,
cb => sub {
warn "Timer 2 at " . AnyEvent->now . "\n";
}
),
AnyEvent->timer (
after => 20,
cb => sub {
warn "Timer 3 at " . AnyEvent->now . "\n";
$cv->send;
}
),
);
$cv->recv;
If you run this you will see something along these lines:
Timer 1 at 1591953440.37264
Timer 2 at 1591953440.37264
Timer 1 at 1591953445.37213
Timer 2 at 1591953447.37533
Timer 1 at 1591953450.36854
Timer 2 at 1591953454.38475
Timer 1 at 1591953455.36656
Timer 3 at 1591953458.37977
So the timers seem to be working on their correct schedules. Now we can add in some pings. I've reduced the numbers here so we don't get bucketloads of output, but the principle should be the same.
#!/usr/bin/env perl
use strict;
use warnings;
use AnyEvent;
use AnyEvent::Ping;
use Data::Dumper;
my @todoList = map { "192.168.100." . $_ } (1 .. 5);
my $cv = AnyEvent->condvar;
my @timers = (
AnyEvent->timer (
after => 2,
interval => 5,
cb => sub {
warn "Timer 1 at " . AnyEvent->now . "\n";
ping_hosts (1, @todoList);
}
),
AnyEvent->timer (
after => 2,
interval => 7,
cb => sub {
warn "Timer 2 at " . AnyEvent->now . "\n";
ping_hosts (2, @todoList);
}
),
AnyEvent->timer (
after => 20,
cb => sub {
warn "Timer 3 at " . AnyEvent->now . "\n";
$cv->send;
}
),
);
$cv->recv;
sub ping_hosts {
my ($num, @hosts) = @_;
my $ping_timeout = 2; #seconds to wait for response
my $ping_attempts = 4; #number of pings to issue,
my $ping_pktsize = 32; #packet size of ping
my $ping = AnyEvent::Ping->new (packet_size => $ping_pktsize);
$ping->timeout ($ping_timeout);
#$cv_ping->begin(sub { shift->send($_[0]) });
foreach my $host (@hosts) {
$ping->ping (
$host,
$ping_attempts,
sub {
print "Pinger $num: $host state:" . Dumper ($_[0]) . "
+\n";
}
);
}
}
I've simplified your ping loop/sub a bit and you'll notice that there is now no counter and importantly no cond_var. This is because you have no need to block here if all you are doing is firing off the pings. Now when we run this it runs to completion with the pings sent on the intended schedule.
This is my first go at using event loops so I am slowly grasping the concepts and how to implement it, but i'm struggling a little, so I'm not even sure if i'm doing this properly.
The concepts are tough - there's no doubt about that. I don't do much functional programming and even less on event loops so it was interesting to approach this from an equally inexpert standpoint. My simplistic approach with these is just not to block unless it's really necessary. That's not always easy to do when you are coming from a procedural background but it has helped me through when tackling this sort of task.
I hope this is useful for you and wish you luck with the rest of your project. | [reply] [d/l] [select] |
|
Re: Implementing AnyEvent::Ping multiple times -- MCE
by Discipulus (Canon) on Jun 15, 2020 at 20:29 UTC
|
Hello mmoorreett
another rope to learn.. but you can be interested in ping_tcp.pl from MCE. The example uses MCE::Hobo which provide its own timeout.
marioroy's effort to support windows is commendable.
L*
There are no rules, there are no thumbs..
Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.
| [reply] [d/l] |
|
cool thanks for that, I haven't seen this one before. i'll check it out.
| [reply] |
Re: Implementing AnyEvent::Ping multiple times
by bliako (Monsignor) on Jun 14, 2020 at 12:05 UTC
|
Perhaps AnyEvent::FastPing can be of help. Alternatively ...
Being unfamiliar, as well, with AnyEvent I could be wrong but how about using a global table (shared among the timer callbacks - is this allowed? It seems OK.) to store the state of each host like: my $state{'192.168.100.1' => [0, ['success at timexxx with packetsxxx', 'failed at ...', ...]]}; where the first array element is a flag indicating whether a ping is in progress right now or not. And the second element stores the history of ping results, possibly in a finite-size data structure (Queue?).
With the above idea and modifying slightly hippo's code:
use strict;
use warnings;
use AnyEvent;
use AnyEvent::Ping;
use Data::Dumper;
# separate the hosts wrt their timer groups
my @todoList1 = map { "192.168.100." . $_ } (1 .. 2);
my @todoList2 = map { "192.168.100." . $_ } (3 .. 4);
# a global hash keyed on IPs, => [0,[]]: no pinging now, =>[1,[]]: pin
+g in progress
# the array ref storing ping history can be a Queue of finite size
my $pingState = { map { $_ => [0,[]] } @todoList1, @todoList2 };
my $cv = AnyEvent->condvar;
my @timers = (
AnyEvent->timer (
after => 2,
interval => 5,
cb => sub {
warn "Timer 1 at " . AnyEvent->now . "\n";
ping_hosts (1, @todoList1);
}
),
AnyEvent->timer (
after => 2,
interval => 7,
cb => sub {
warn "Timer 2 at " . AnyEvent->now . "\n";
ping_hosts (2, @todoList2);
}
),
AnyEvent->timer (
after => 20,
cb => sub {
warn "Timer 3 at " . AnyEvent->now . "\n";
$cv->send;
}
),
);
$cv->recv;
print "DONE.\n".Dumper($pingState);
sub ping_hosts {
my ($num, @hosts) = @_;
my $ping_timeout = 2; #seconds to wait for response
my $ping_attempts = 4; #number of pings to issue,
my $ping_pktsize = 32; #packet size of ping
my $ping = AnyEvent::Ping->new (packet_size => $ping_pktsize);
$ping->timeout ($ping_timeout);
#$cv_ping->begin(sub { shift->send($_[0]) });
foreach my $host (@hosts) {
if( $pingState->{$host}->[0] == 1 ){
print "ping in progress already to host '$host', skipp
+ing till the next timer ...\n";
next
}
$pingState->{$host}->[0] = 1; # set the "ping in progress" fla
+g
$ping->ping (
#mock_ping( # emulate the ping
$host,
$ping_attempts,
sub {
print "Pinger $num: $host state:" . Dumper ($_[0]) . "
+\n";
# store the ping results in global history and remove
+the "ping-in-progress" flag
push @{$pingState->{$host}->[1]}, $_[0];
$pingState->{$host}->[0] = 0; # ping finished
},
# mock_ping extra parameters
# $ping_timeout,
# 50/100 # probability of a failed ping
);
}
}
# code below is
# optional for testing without ping'ing
sub mock_ping {
my ($host, $attempts, $cb,
$timeout, $failure_prob) = @_;
# no $attempts, just fail or succeed so many % of times
my $started = AnyEvent->time;
if( rand()<$failure_prob ){
# simulate a failed ping with lots of delay
# so as to coincide with next timer event
AEsleep($timeout*3);
$cb->([$host, "$started -> fail -> ".AnyEvent->time]);
return 0;
} else {
# a successful ping safely within the timeout
AEsleep($timeout/2); # simulate a ping
$cb->([$host, "$started -> succ -> ".AnyEvent->time]);
return 1; # success ping
}
}
# this is the closest I came to emulating spawning a blocking
# ping. Note that using Perl's sleep() is no-go with AnyEvent's timers
# AFAIU, re: "It is usually a mistake to intermix alarm and sleep call
+s"
# and sadly there is no AE::sleep() AFAIK
sub AEsleep($) { `sleep $_[0]` }
btw, in my Linux the pinging code needs to be run as root.
bw, bliako | [reply] [d/l] [select] |
|
I did try AnyEvent::FastPing but it won't compile via cpan...
thanks for the tip, i'll give it a go
| [reply] |
|
|