Hey all, I seem to have a memory leak in my threaded script... I haven't done much thread work so it is probably an oversight on my part, I'd appreciate a look.
This is a simplified version that exhibits the problem, I am trying a classic method of one worker filling a queue with multiple workers reading from the queue and managing the data, my total dataset is over a million records so I am breaking it into 10000 record chunks, but it would seem they are hanging around in memory. This example just makes data forever but clearly shows that the data is sticking around when you watch top.
I am using Perl 5.8.5, threads 1.73, threads::shared 1.29 and Thread::Queue 2.11, all on Linux
use threads;
use threads::shared qw(share);
use Thread::Queue;
use Thread::Semaphore;
my $q : shared = Thread::Queue->new();
my $loading : shared = 1;
my $threads : shared = 0;
my $thr = threads->create( \&load_queue, $q, \$loading );
$thr->detach();
$|++;
my @threads;
for ( 1 .. 6 ) {
my $thr = threads->create( \&{type_issues}, $q, \$loading, $thread
+s );
push @threads, $thr;
}
$_->join() for @threads;
sub load_queue {
my $q = shift;
my $loading = shift;
my $i = 1;
my $ary = &share( [] );
while ( my $v = ["this is a test" x 250] ) {
# print "$v->[0]\n";
push @$ary, $v->[0];
unless ( $i++ % 10000 ) {
print "$i\n";
$q->enqueue($ary);
$ary = &share( [] );
while ( $q->pending() > 20 ) { # try not to eat tooo mu
+ch memory
select(undef,undef,undef,.1);
}
}
}
$q->enqueue($ary) if @$ary;
$$loading = 0;
}
sub type_issues {
my ( $q, $loading ) = @_;
#my $DB = IDC::Data->new( bes => 'webdev' );
print "Type issues\n";
my $issues;
while ( ( $issues = $q->dequeue() ) ) {
print "Got ".@$issues." - ".$q->pending()."\n";
select(undef,undef,undef,.1);
}
}
- Ant
- Some of my
best work - (1 2 3)