ecuguru has asked for the wisdom of the Perl Monks concerning the following question:

Hi,
I've got a loop that goes through a directory of huge xml files and after some period of time, will eventually die with an out of memory error. While my program is running, the memory usage on the stat for it continues to go up until I get an out of memory error. It runs until it runs out of memory. I restart the application, and it starts running and parses the file it just died on previously.
perl(8145) malloc: *** vm_allocate(size=8421376) failed (error code=3) perl(8145) malloc: *** error: can't allocate region perl(8145) malloc: *** set a breakpoint in szone_error to debug Out of memory!
I'm guessing that its my usage of XML Dom. For each xml file in a directory, I create a new parser and a new doc. But I haven't found any code that will let me delete or free the parser or the doc the parser is parsed into. Could that be the problem? If so, how do I free it, and which should I free?
use XML::DOM; my @files = <$datapath/*.*>; foreach my $file (@files) { my $parser = XML::DOM::Parser->new(); my $doc = $parser->parsefile($file); ##I parse the documents }
ANSWER: Looks like its $doc->dispose.

Replies are listed 'Best First'.
Re: XML Out of Memory
by samtregar (Abbot) on Jul 19, 2007 at 04:39 UTC
    Glad you found your answer - but I thought I'd mention that XML::DOM is not a great tool for dealing with very large XML files. XML::SAX is a better fit - it's supports streaming parsers (like XML::SAX::ExpatXS) which means they doesn't build a tree structure in memory. Depending on your task this can mean your app runs faster and uses far less memory. Used correctly an XML::SAX handler should be able to process arbitrarily large XML files.

    -sam

      &tThanks!
      I thought I would leave the question up with the answer and see if anyone like you who had more experience in this area would be nice enough to contribute other ideas... I'll definintely take a look at it.
      thanks again!