I have parsed huge XML files up to 2GB and I can tell you it takes time, especially if you take an XML-ish approach, i.e. use a XML parser. Of course any DOM-approach is out of the question, loading such a document in memory is asking for trouble.

It depends of course on the complexity of the XML file, the hardware configuration etc. etc. but if you parse huge XML files you basically have to be patient. Don’t expect miracles from other modules, there is no silver bullet when parsing huge XML files.

In my situation parsing the file and updating some nodes took over 1 hour for a 1GB file with XML::Twig. For my requirements this was not sufficient.

Some ideas most of which I explored in the past:

• Rethink the problem!, maybe it’s possible to decrease the size of the XML files. I mean XML files of 3-4GB do sound a bit weird/large.
• Map the XML structure onto a relational database, do a bulk load and let the DB do the work for you.
• Choose a non-XML approach, instead of parsing the file with a parser you might opt for handcrafted Perl solution.
• Take a look at other environments, I got some very good performance out of Xalan using SAX. There is C++ version if you’re allergic to Java;)
• There are native XML databases with good performance but this most likely means spending money!

HTH,
dHarry

You can find PPMs for XML::Twig in Kobe's repository: http://cpan.uwinnipeg.ca/module/XML::Twig.

That said, I don't know how you load a 4gb file in XML::DOM, how much memory do you have on that machine? 40gb? With XML::Twig you can process parts of the XML (twigs as opposed to the whole tree ;--) so you should be able to keep memory usage lower, maybe much lower, depending on what you need to do, which should speedup processing. But if you can load the entire tree in memory and you can install libxml2, then you can also try using XML::LibXML, porting the XML::DOM code would be much easier in this case, and XML::LibXML is much faster than XML::DOM.

Hmm Yes. My memory is 2Gb only.:(

My Xml contains Multiple blocks of data . one block is like below.

<SigData>
<KVPair>
<Key>eb08f9990ae6545f9ea625412c71f24f7bf007ed</Key>
<Value>c73df5228c35c419f884ba9571310cd7</Value>
</KVPair>
</SigData>


i need to load these elements <key>,<value> of the tree into these arrays like

my @keys = getValuesFromPath($sigData ,"/SigData/KVPair/Key");
my @values = getValuesFromPath($sigData ,"/SigData/KVPair/Value");

So you want me to use XML::LibXML also along with XML::Twig???

I was just surprised that you could use XML::DOM at all on files of that size. And it looks like you can't actually, a 1gb XML file would take at least 8gb in memory using XML::DOM. So it might be interesting to know how you did it. What I meant was that if you had been able to do it, by throwing large amounts of memory at the problem, then XML::LibXML would have been an option.

With XML::Twig you can very easily extract the k/v pairs:

my $t= XML::Twig->new( twig_roots => 
  { SigData => sub { push @keys, $_->field( 'Key');
                     push @values, $_->field( 'Value');
                     $_->purge;
                   } 
  },
                         )
                   ->parsefile("my_big_fat_xml_file.xml");
[download]

Of course the @keys and @values arrays are going to be huge too, so you might still want to add a few GB of RAM to your machine, but at least the XML structure will never take up more than a few bytes.

Other possible options are XML::Rules (I expect jenda to show up and give you an example as soon as he wakes up, and maybe the new XML::Reader, which seems quite appropriate. XML::LibXML's pull mode might also be appropriate, but I have never used it so I can't comment on it.