Hello monks, I'm using XML::Twig on Windows 7 (Strawberry Perl v5.16.3) to parse some pretty large XML files (e.g. ~800MB) and getting the old "out of memory!" error message. The script works fine on smaller files. Probably a dumb question, but am I just SOL? Is there any way to slurp through the file in manageable chunks? I'm using a remote server that I don't own so adding more memory is not an option.

(also tried XML::Simple)

Here's my code if it helps, but it's not really doing anything complicated or interesting.

use XML::Twig;
use strict;

my $file = shift;

# set up the XML parser:
my $twig= XML::Twig->new( 
  comments => 'keep',
  twig_handlers => { row => \&row_processing },
        pretty_print => 'indented',
);

print " parsing $file...\n";

$twig->parsefile($file);
$twig->purge;

sub row_processing {
 my($twig, $rows)= @_;
 print "a row\n";
}

[download]

thanks -- Scott

In reply to out of memory! parsing large XML file by slugger415

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.