out of memory issue while creating very large xml file using XML::Simple

perlCrazy has asked for the wisdom of the Perl Monks concerning the following question:

How we can create very large xml file(1 GB) using XML::Simple module.
variable $tableDetailsRef is ref of hash, will hold very large blob. While using with xmlout causing memory issue. Host is RHEL6 and memory available is 36 GB. I am using below code:

#this variable will hold very large complex data structure. 
my $tableDetailsRef = {};

my $xmlFile = "test_name.xml";
my $targetNameStr = "target-name";
#Creating object for xml file
my $xmls = XML::Simple->new( RootName   => undef,
                           ForceArray => ['tables'],
                           KeyAttr    => { table => 'id' },

);
my $contents = {};
$contents->{dspl}->{targetNamespace} = "string_test";

################################################
if(defined $tableDetailsRef){
    push @{ $contents->{dspl}->{'tables'} },
      {
         table => $tableDetailsRef,
      };
}
    
####################################################
##Generate the xml file
eval {
       $xmls->XMLout( $contents,
                      xmldecl    => '<?xml version="1.0" encoding="UTF
+-8" standalone="no" ?>',
                      OutputFile => $xmlFile
   );
};
    if($@) {
        print "Exception: Could not xml file : $xmlFile, Error-$@");
    }
[download]

Comment on out of memory issue while creating very large xml file using XML::Simple Download Code

Replies are listed 'Best First'.
Re: out of memory issue while creating very large xml file using XML::Simple by Discipulus (Canon) on Jul 22, 2014 at 11:27 UTC
short answer of two words: XML::Twig and Iterators. HtH L* There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.	[reply] [d/l]
Re: out of memory issue while creating very large xml file using XML::Simple by Your Mother (Archbishop) on Jul 22, 2014 at 17:12 UTC
From the docs for XML::Simple– What isn't XML::Simple good for? The main limitation of XML::Simple is that it does not work with 'mixed content' (see the next question). If you consider your XML files contain marked up text rather than structured data, you should probably use another module. If you are working with very large XML files, XML::Simple's approach of representing the whole file in memory as a 'tree' data structure may not be suitable. XML::Twig and XML::LibXML are both better options for most XML processing, not just large files.	[reply]
Re: out of memory issue while creating very large xml file using XML::Simple by locked_user sundialsvc4 (Abbot) on Jul 22, 2014 at 14:42 UTC
I’ll presume that you are intending by the above to create a minimalistic example. But, we need details here: how large is this data structure? Are you running a 64-bit version of Perl? Do you have any idea as to the memory-footprint of this application prior to the step that generates the XML file? Is there any possibility at all of self-referential “loops” within this data structure, which might cause an algorithm to iterate endlessly? (Test::Memory::Cycle, and Devel::Cycle, can automatically look for these.) Obviously, it is unreasonable to expect that a program running on a 32GB machine, dealing with a data structure that is thought to be about 1GB, and running 64-bit software in a 64-bit environment, would be encountering such issues. (Whereas, BTW, if the actual system were 32-bit, the total memory space would be less than 2GB, and overflow is a distinct possibility.) So, we really don’t have enough solid details here with which to offer many solutions . . . yet. I’m not yet persuaded that we should finger this particular package. Rather, I suspect that there is something else going on that we don’t yet see.
Re^2: out of memory issue while creating very large xml file using XML::Simple by perlCrazy (Monk) on Jul 22, 2014 at 16:48 UTC
Thanks for response. When we increased the memory limit from 2gb to 20 GB for a process it went fine. datastructure was taking around 30 MB. we broke the datastructure and wrote to disk using storable module. main issue while writing xml file using XML::Simple. xmlOut method expects all contents in one data structure. it would have been better if we can write like streaming or inside loop. is there any way to create xml file in append mode using xml::simple?	[reply]

What isn't XML::Simple good for?