Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling

Re^4: XML::Twig loves to eat my memory

by carcus88 (Acolyte)
on Jul 22, 2010 at 21:05 UTC ( [id://850912]=note: print w/replies, xml ) Need Help??

in reply to Re^3: XML::Twig loves to eat my memory
in thread XML::Twig loves to eat my memory

Here is a much more minimal example. Both times I tried this once without the extra purge() and once with. Both times memory was at 166MB after the dispose().
#!/usr/bin/perl -w use strict; use XML::Twig; my $inFile = '../data/2009/data_verified/abms_dialogb_10000000_1009999 +9.xml'; if ( ! $inFile ) { die("No input file specified"); } if ( ! -f $inFile ) { die("file '$inFile' not found"); } process($inFile); exit 0; # # Process the file # sub process { $inFile =~ /data_(\d+)_(\d+)/; my $t= new XML::Twig( TwigHandlers=> { BIOG => \&BIOG }, ); $t->parsefile( $inFile ); $t->purge(); # Will purge work? $t->dispose(); # Try to Free memory but does not work... } # # BIOG is XML element we are triggering # sub BIOG { my ($t, $BIOG)= @_; if ( ! checkBiog($BIOG->field('BIOG_NBR')) ) { print "Missing ". $BIOG->field('BIOG_NBR') . "\n"; } $t->purge(); # Tell XML::Twig to dispo of the rest of the tree we + don't care about return 1; } # # Check database for ID # sub checkBiog { my ($biog) = @_; return 1; }
I will try to work on generating an example XML file. The current XML file is licensed data that I cannot share.

Replies are listed 'Best First'.
Re^5: XML::Twig loves to eat my memory
by almut (Canon) on Jul 22, 2010 at 21:20 UTC
    ...memory was at 166MB after the dispose()

    Does the memory usage increase another ~166MB if you call process() a second time with the same file?  (Your original post sounds like it does, but just to be sure...)

    The reason I'm asking is because memory might not be returned to the OS immediately, but just freed for reuse Perl-internally.

      Yes it continues to increase
      Size after first process() 
      167,472 MB
      Size after second process()
      316,212 MB

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://850912]
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others admiring the Monastery: (2)
As of 2024-04-16 14:47 GMT
Find Nodes?
    Voting Booth?

    No recent polls found