wazoox has asked for the wisdom of the Perl Monks concerning the following question:
Dear brethren,
I'm looking for an elegant way of reading data from a file on one hand, and serve it (preferably through a websocket) on the other hand. So far I managed to solve elegantly the problem of reading the output from several programs simultaneously by using on of the big powers of Unix : named pipes. However now I'm stuck on the next part: serving data. As reading from a pipe is blocking, and so is listening to a socket, I don't know how to get out of this.
So here's the code:
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
use Net::WebSocket::Server;
use JSON;
my %data;
my $json;
# Net::WebSocket::Server->new(
# listen => 8080,
# on_connect => sub {
# my ($serv, $conn) = @_;
# $conn->on(
# handshake => sub {
# my ($conn, $handshake) = @_;
# },
# utf8 => sub {
# my ($conn) = @_;
# $_->send_utf8($json) for $conn->server->connections;
# },
# );
# },
# )->start;
open my $fh, '<', 'toto1' or die "can't open fifo toto1: $?";
while ( my $line = <$fh> ) {
my @list = ( split /\s+/, $line );
# skip empty lines
next if $#list <= 1;
shift @list if $list[0] eq '';
# test for iostat header
next if $list[0] eq 'avg-cpu:';
next if $list[0] eq 'Device:';
# test for vmstat header
next if $list[0] eq 'procs';
if ( $#list == 11 ) {
(
undef,
$data{io}{ $list[0] }{rrqms},
$data{io}{ $list[0] }{wrqms},
$data{io}{ $list[0] }{reads},
$data{io}{ $list[0] }{writes},
$data{io}{ $list[0] }{rsecs},
$data{io}{ $list[0] }{wsecs},
$data{io}{ $list[0] }{avgrqsz},
$data{io}{ $list[0] }{avgqusz},
$data{io}{ $list[0] }{await},
$data{io}{ $list[0] }{svctm},
$data{io}{ $list[0] }{util}
) = @list;
} elsif ( $#list == 15 ) {
(
$data{vm}{r}, $data{vm}{b}, $data{vm}{swpd},
$data{vm}{free}, $data{vm}{buff}, $data{vm}{cache},
$data{vm}{swapin}, $data{vm}{swapout}, $data{vm}{ioin},
$data{vm}{ioout}, $data{vm}{sysin}, $data{vm}{syscs},
$data{vm}{user}, $data{vm}{sys}, $data{vm}{idle},
$data{vm}{wait}
) = @list;
} elsif ( $#list == 5 ) {
$data{vm}{nice} = $list[1];
}
$json = encode_json( \%data);
sleep 1;
}
Of course the script work by commenting out one of the two loops, either the websocket one, or the file reading one. To use this script, create a named pipe mkfifo toto1 run continuously vmstat and/or iostat and redirect their output to the pipe: vmstat 3 |tee toto1 and iostat -x 3 | tee toto1.
I don't know how to get any further from there... maybe golang? :)
Re: Simultaneously reading from a file and serving data
by kennethk (Abbot) on Feb 27, 2014 at 19:11 UTC
|
| [reply] |
Re: Simultaneously reading from a file and serving data
by davido (Cardinal) on Feb 27, 2014 at 19:29 UTC
|
This isn't addressing your question, but one thing I noticed in your code is your verbose method of initializing %data. Note:
use Test::More;
use constant MY_KEYS =>
qw( rrqms wrqms reads writes rsecs wsecs avgrqsz avgqusz await svctm
+ util );
my $key = 'asdf';
my @list = ( 'a' .. 'l' );
my %data;
(
undef, $data{io}{$key}{rrqms},
$data{io}{$key}{wrqms}, $data{io}{$key}{reads},
$data{io}{$key}{writes}, $data{io}{$key}{rsecs},
$data{io}{$key}{wsecs}, $data{io}{$key}{avgrqsz},
$data{io}{$key}{avgqusz}, $data{io}{$key}{await},
$data{io}{$key}{svctm}, $data{io}{$key}{util}
) = @list;
my %data2;
( undef, @{ $data2{io}{$key} }{ +MY_KEYS } ) = @list;
my %data3;
@{ $data3{io}{$key} }{ +MY_KEYS } = @list[ 1 .. $#list ];
is_deeply \%data, \%data2, "Assigning an array to a slice.";
is_deeply \%data, \%data3, "Assigning a slice to a slice.";
done_testing();
__END__
__OUTPUT__
ok 1 - Assigning an array to a slice.
ok 2 - Assigning a slice to a slice.
1..2
| [reply] [d/l] [select] |
Re: Simultaneously reading from a file and serving data
by thargas (Deacon) on Feb 27, 2014 at 19:29 UTC
|
Have you looked at IO::Select? It will allow you to have a loop which is "simultaneously" waiting for input from multiple sources and waiting for outputs to become ready for write and allowing for timeouts. Or you could use the underlying select call. | [reply] |
|
select is definitely how I would do it, for several key reasons: - It is architecturally uncomplicated: the process is actually doing only one thing at a time, even though it is unpredictable what it might do next.
- You never have to worry about things being incomplete ... about accidentally sending someone a partial-message that you haven’t finished reading yet.
- You can easily prioritize what it is doing, according to the business requirement.
- Messy timing issues are generally avoided completely.
The program itself is really a dispatcher, connected to everything else with nice, flexible pipes and sockets. It goes to sleep, wakes up with a “honey-do list,” does its chores in whatever sequence it best sees fit, then goes back to sleep again. As long as none of these things take a significant time to actually do, esp. relative to the others, it all works great.
| [reply] |
Re: Simultaneously reading from a file and serving data
by BrowserUk (Patriarch) on Feb 27, 2014 at 21:41 UTC
|
Completely untested, this should be close:
#!/usr/bin/perl
use strict;
use warnings;
use threads:
+ ## Added
use threads::shared;
+ ## Added
use Data::Dumper;
use Net::WebSocket::Server;
use JSON;
my %data;
my $json :shared;
+ ## Modded
async {
+ ## Added
Net::WebSocket::Server->new(
listen => 8080,
on_connect => sub {
my ($serv, $conn) = @_;
$conn->on(
handshake => sub {
my ($conn, $handshake) = @_;
},
utf8 => sub {
my ($conn) = @_;
lock $json;
+ ## Added
$_->send_utf8($json) for $conn->server->connection
+s;
},
);
},
)->start;
}->detach;
+ ## Added
open my $fh, '<', 'toto1' or die "can't open fifo toto1: $?";
while ( my $line = <$fh> ) {
my @list = ( split /\s+/, $line );
# skip empty lines
next if $#list <= 1;
shift @list if $list[0] eq '';
# test for iostat header
next if $list[0] eq 'avg-cpu:';
next if $list[0] eq 'Device:';
# test for vmstat header
next if $list[0] eq 'procs';
if ( $#list == 11 ) {
(
undef,
$data{io}{ $list[0] }{rrqms},
$data{io}{ $list[0] }{wrqms},
$data{io}{ $list[0] }{reads},
$data{io}{ $list[0] }{writes},
$data{io}{ $list[0] }{rsecs},
$data{io}{ $list[0] }{wsecs},
$data{io}{ $list[0] }{avgrqsz},
$data{io}{ $list[0] }{avgqusz},
$data{io}{ $list[0] }{await},
$data{io}{ $list[0] }{svctm},
$data{io}{ $list[0] }{util}
) = @list;
} elsif ( $#list == 15 ) {
(
$data{vm}{r}, $data{vm}{b}, $data{vm}{swpd},
$data{vm}{free}, $data{vm}{buff}, $data{vm}{cache},
$data{vm}{swapin}, $data{vm}{swapout}, $data{vm}{ioin},
$data{vm}{ioout}, $data{vm}{sysin}, $data{vm}{syscs},
$data{vm}{user}, $data{vm}{sys}, $data{vm}{idle},
$data{vm}{wait}
) = @list;
} elsif ( $#list == 5 ) {
$data{vm}{nice} = $list[1];
}
{ ##
+ Added
lock $json; ##
+ Added
$json = encode_json( \%data);
} ##
+ Added
sleep 1;
}
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] [d/l] |
|
That mostly works :) I'm trying to get the client part working, and I'll share the lot, because a web-based system real-time system monitor may probably be of some use to others.
| [reply] |
Re: Simultaneously reading from a file and serving data
by karlgoethebier (Abbot) on Feb 27, 2014 at 20:34 UTC
|
BTW, in this context it might be interesting to show how i went over this bridge (AKA "Install Perl with threads" or so) using perlbrew:
karls-mac-mini:~ karl$ perlbrew list
perl-5.16.2
perl-5.16.3
perl-5.17.7
perl-5.18.0
perl-5.18.1
* perl-5.18.2
karls-mac-mini:~ karl$ perlbrew install --as perl-5.18.2threads -Duse
+threads perl-5.18.2
Installing /Users/karl/perl5/perlbrew/build/perl-5.18.2 into ~/perl5/p
+erlbrew/perls/perl-5.18.2threads
This could take a while. You can run the following command on another
+shell to track the status:
tail -f ~/perl5/perlbrew/build.perl-5.18.2.log
karls-mac-mini:~ karl$ perlbrew list
perl-5.16.2
perl-5.16.3
perl-5.17.7
perl-5.18.0
perl-5.18.1
* perl-5.18.2
perl-5.18.2threads
karls-mac-mini:~ karl$ perlbrew switch perl-5.18.2threads
karls-mac-mini:~ karl$ perlbrew list
perl-5.16.2
perl-5.16.3
perl-5.17.7
perl-5.18.0
perl-5.18.1
perl-5.18.2
* perl-5.18.2threads
karls-mac-mini:~ karl$ perl -Mthreads
^C
Seems to be pretty cool.
Regards, Karl
«The Crux of the Biscuit is the Apostrophe»
| [reply] [d/l] |
Re: Simultaneously reading from a file and serving data
by grondilu (Friar) on Feb 27, 2014 at 17:45 UTC
|
I'm no expert but: can't you just use a fork? I haven't used it for a while but IIRC it's actually not very hard to use and there are several nice examples in perlipc.
Something like:
if (my $pid = fork()) {
# network stuff
} else {
# file stuff
}
| [reply] [d/l] |
|
The problem with a fork is that the %data won't be shared between processes...
| [reply] |
|
WARNING: I'm afraid this message is actually irrelevant. See EDIT note at the end.
Well, you can have your two processes communicate with yet another named pipe. I really think you should (re-)read perlipc. There are lots of examples for this kind of stuff. Check out the open() command for instance. You can use it to fork and create a named pipe in the same time. See the "Safe Pipe Opens" section.
Maybe something like this:
if (my $pid = open(JSON, '|-') {
# database stuff
print JSON $json;
} else {
# this is the child fork.
# STDIN here is actually JSON in the other fork.
# network stuff
my $json = <>;
# some more network stuff I guess
}
EDIT I've just realized this wont solve your issue since the child process would block waiting for data from the parent process. It can't do that *AND* act as a network server at the same time. It only moves the blocking problem from one place to an other. So I'm sorry, I don't know. | [reply] [d/l] |
|
|