HTML::TableExtract Memory Usage

corpx has asked for the wisdom of the Perl Monks concerning the following question:

Here's my code

#!/usr/bin/perl

use WWW::Mechanize;
use HTML::TableExtract qw(tree);;
use strict;

start();    


sub start {
    my $te = HTML::TableExtract -> new(  );    
    my $agent = WWW::Mechanize->new( stack_depth => 0);
    $agent-> agent_alias( 'Windows IE 6' );
    $agent->get('http://www.perlmonks.org/?');
    my $page = $agent->content;
    
    
    for (my $c =0 ; $c < 500; $c++)
    {
        $te->parse ($page);
        print "c is $c\n";
    }
}
exit();
[download]

Every time it executes $te->parse($page), the memory used by the program increases. I know HTML::Tree has delete() to clear up the memory after using it, but how can I free up the memory used by TableExtract?

Comment on HTML::TableExtract Memory Usage Select or Download Code

Replies are listed 'Best First'.
Re: HTML::TableExtract Memory Usage by Corion (Patriarch) on Sep 02, 2010 at 19:51 UTC
It seems that other people also experienced this, as this bug report on RT tells. No immediate solution is suggested, but maybe you can see how the HTML::TreeBuilder deletion should be done better.	[reply]
Re^2: HTML::TableExtract Memory Usage by corpx (Acolyte) on Sep 02, 2010 at 21:19 UTC
That sucks. I've been pouring over it for a while, but no luck yet.	[reply]
Re: HTML::TableExtract Memory Usage by Anonymous Monk on Sep 03, 2010 at 00:37 UTC
Try `$te->tree->delete;` or `sub HTML::TableExtract::Table::DESTROY { eval { $_[0]->tree->delete; $_[0]->tree( undef ); }; return; }` [download] or `my $te = HTML::TableExtract -> new( ); $te->parse ($page); $te->eof; print "$te c is $c\n"; eval { for( $te->tables ){ $_->tree->delete ; $_->tree( undef ); } $te->tree->delete; $te->tree( undef ); undef $te; 1; } or warn $@;` [download] it might work	[reply] [d/l] [select]
Re^2: HTML::TableExtract Memory Usage by Anonymous Monk on Sep 03, 2010 at 00:49 UTC
Just tried all 3 of the above scenarios, but the memory keeps ballooning :/	[reply]
Re^3: HTML::TableExtract Memory Usage by Anonymous Monk on Sep 03, 2010 at 04:06 UTC
I was afraid of that. There are too many circular references in that module, ie `sub _reset_state { my $self = shift; $self->{_cdepth} = -1; $self->{_tablestack} = []; $self->{_tables} = {}; $self->{_ts_sequential} = []; $self->{_counts} = []; $self->{_in_a_table} = 0; } ... grid => [], translation => [], hrow => [], order => [], children => [],` [download] Would have to break all those circular references in DESTROY, recursively	[reply] [d/l]
Re: HTML::TableExtract Memory Usage by Anonymous Monk on Sep 04, 2010 at 14:30 UTC
This leaks practically nothing Read more... (1478 Bytes) on my machine it starts at 17mb/14mb VM, and by the 500th iteration its at 18mb/16mb. My guess is this very tiny leak is probably in HTML::Parser.	[reply] [d/l]
Re: HTML::TableExtract Memory Usage by Anonymous Monk on Sep 04, 2010 at 07:03 UTC
Its related to HTML::ElementTable, it leaks memory `perl -MHTML::ElementTable -e " for(;;){ my $c = HTML::ElementTable->ne +w; warn $c; undef $c; } "` [download]	[reply] [d/l]
Re^2: HTML::TableExtract Memory Usage by Anonymous Monk on Sep 04, 2010 at 07:33 UTC
This also leaks `perl -MHTML::ElementTable -e " for(;;){ my $c = HTML::ElementTable->ne +w; warn $c; $c->delete; undef $c; } "` [download]	[reply] [d/l]


Clear questions and runnable code get the best and fastest answer
	PerlMonks