Re: out of memory issue while creating very large xml file using XML::Simple

I’ll presume that you are intending by the above to create a minimalistic example. But, we need details here: how large is this data structure? Are you running a 64-bit version of Perl? Do you have any idea as to the memory-footprint of this application prior to the step that generates the XML file? Is there any possibility at all of self-referential “loops” within this data structure, which might cause an algorithm to iterate endlessly? (Test::Memory::Cycle, and Devel::Cycle, can automatically look for these.)

Obviously, it is unreasonable to expect that a program running on a 32GB machine, dealing with a data structure that is thought to be about 1GB, and running 64-bit software in a 64-bit environment, would be encountering such issues. (Whereas, BTW, if the actual system were 32-bit, the total memory space would be less than 2GB, and overflow is a distinct possibility.) So, we really don’t have enough solid details here with which to offer many solutions . . . yet. I’m not yet persuaded that we should finger this particular package. Rather, I suspect that there is something else going on that we don’t yet see.

Replies are listed 'Best First'.
Re^2: out of memory issue while creating very large xml file using XML::Simple by perlCrazy (Monk) on Jul 22, 2014 at 16:48 UTC
Thanks for response. When we increased the memory limit from 2gb to 20 GB for a process it went fine. datastructure was taking around 30 MB. we broke the datastructure and wrote to disk using storable module. main issue while writing xml file using XML::Simple. xmlOut method expects all contents in one data structure. it would have been better if we can write like streaming or inside loop. is there any way to create xml file in append mode using xml::simple?	[reply]

Replies are listed 'Best First'.

Re^2: out of memory issue while creating very large xml file using XML::Simple
by perlCrazy (Monk) on Jul 22, 2014 at 16:48 UTC

Thanks for response. When we increased the memory limit from 2gb to 20 GB for a process it went fine. datastructure was taking around 30 MB. we broke the datastructure and wrote to disk using storable module. main issue while writing xml file using XML::Simple. xmlOut method expects all contents in one data structure. it would have been better if we can write like streaming or inside loop. is there any way to create xml file in append mode using xml::simple?

[reply]