I have parsed huge XML files up to 2GB and I can tell you it takes time, especially if you take an XML-ish approach, i.e. use a XML parser. Of course any DOM-approach is out of the question, loading such a document in memory is asking for trouble.

It depends of course on the complexity of the XML file, the hardware configuration etc. etc. but if you parse huge XML files you basically have to be patient. Don’t expect miracles from other modules, there is no silver bullet when parsing huge XML files.

In my situation parsing the file and updating some nodes took over 1 hour for a 1GB file with XML::Twig. For my requirements this was not sufficient.

Some ideas most of which I explored in the past:

• Rethink the problem!, maybe it’s possible to decrease the size of the XML files. I mean XML files of 3-4GB do sound a bit weird/large.
• Map the XML structure onto a relational database, do a bulk load and let the DB do the work for you.
• Choose a non-XML approach, instead of parsing the file with a parser you might opt for handcrafted Perl solution.
• Take a look at other environments, I got some very good performance out of Xalan using SAX. There is C++ version if you’re allergic to Java;)
• There are native XML databases with good performance but this most likely means spending money!

HTH,
dHarry