I frequently suggest Thread::Queue for pooled thread applications, but in addition to some non-queue-like behavioural cruft, that module has no way of auto-limiting the size of the queue. That means it is all too easy to populate the queue at a rate far in excess of the pool's abilities to process those entries. And that can lead to excessive memory consumption.
Here is an implementation of a shared queue to address that deficiency:
#! perl -slw
use strict;
package Q;
use threads;
use threads::shared;
use constant {
NEXT_WRITE => -2,
N => -1,
};
sub new {
# warn "new: @_\n";
my( $class, $Qsize ) = @_;
$Qsize //= 3;
my @Q :shared;
$#Q = $Qsize;
@Q[ NEXT_WRITE, N ] = ( 0, 0 ); ## nextWrite, N
# warn sprintf "new: size %d\n\n", scalar @Q;
return bless \@Q, $class;
}
sub nq {
# warn "nq: @_\n";
my $self = shift;
lock @$self;
for( @_ ) {
cond_wait @$self until $self->[ N ] < ( @$self-2 );
$self->[ $self->[ NEXT_WRITE ]++ ] = $_;
++$self->[ N ];
$self->[ NEXT_WRITE ] %= ( @$self - 2 );
cond_signal @$self;
}
}
sub dq {
# warn "dq: @_\n";
my $self = shift;
lock @$self;
cond_wait @$self until $self->[ N ] > 0;
my $p = $self->[ NEXT_WRITE ] - $self->[ N ]--;
$p += @$self -2 if $p < 0;
my $out = $self->[ $p ];
cond_signal @$self;
return $out;
}
sub n {
my $self = shift;
# lock @$self;
return $self->[ N ];
}
sub _state {
no warnings;
my $self = shift;
lock @$self;
return join '|', @{ $self };
}
return 1 if caller;
Criticisms and comments on the implementation are welcome, but what I'd really like is for people to post what tests they would implement for this module, and how they would implement them.
It's a big ask I know, and there is a not-so-hidden agenda. Anyone prepared to step up?
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.