in reply to XML::Simple hangs

'Hang' is such an ugly word :-) On my Athlon 2600 laptop it took nearly 2 hours to complete, but it did eventually finish.

I'm guessing that on your system XML::Simple is using XML::Parser. Internally, XML::Simple does something like this:

my $xp = XML::Parser->new(Style => 'Tree'); my $tree = $xp->parse($document);

and then proceeds to reduce $tree into something simpler.

If you try that snippet on your document, then I suspect you'll see similarly long processing times. I'm not sure why expat is so slow processing your XML, but it is rather 'unusual' XML.

Your original document had over 100,000 lines. I benchmarked passing shorter versions of the document through different parser modules using XML::Simple. (If you install XML::SAX, you can chose alternative parsers, by assigning a parser module name to $XML::Simple::PREFERRED_PARSER). I would caution against reading anything at all into these results in a general sense but in the case of your specific data they are interesting:

Parser5000 lines10000 lines15000 lines20000 lines25000 lines
XML::Parser84092169267
XML::SAX::Expat84095173272
XML::SAX::ExpatXS42766121191
XML::SAX::PurePerl920293949
XML::LibXML::SAX::Parser11111

As you can see, the run times for all the expat based parsers are increasing exponentially with the size of the file. The PurePerl parser is coping amazingly well and its run times are only increasing geometricallylinearly. The libxml based SAX parser is the clear winner in this race with run times so short I can't see an increase at all.

Just to illustrate how unusual your XML is compared to 'normal' XML, I changed the 20,000 line file so that instead of looking like this...

<data> SUQzAgAAAAAQaVRUMgAAEQBrZWVwIFlhIEhlYWQgVXAAVFAxAAAUAEhhcmxlbSB0aG cwBUQUwAABQARnJlZSBBZ2VudC9EZWMuIDA0AFRSSwAABQAzLzMAVEVOAAANAGlUdW AENPTQAAaABlbmdpVHVuTk9STQAgMDAwMDA0NUEgMDAwMDAwMDMgMDAwMDMwRjAgMD MDAwMENBNTggMDAwMkNDNkQgMDAwMDgyM0MgMDAwMDgwMEQgMDAwNDAxODIgMDAwND ... </data>

it looked like this ...

<data> <line>SUQzAgAAAAAQaVRUMgAAEQBrZWVwIFlhIEhlYWQgVXAAVFAxAAAUAEhhcmxlbSB0 +aG</line> <line>cwBUQUwAABQARnJlZSBBZ2VudC9EZWMuIDA0AFRSSwAABQAzLzMAVEVOAAANAGlU +dW</line> <line>AENPTQAAaABlbmdpVHVuTk9STQAgMDAwMDA0NUEgMDAwMDAwMDMgMDAwMDMwRjAg +MD</line> <line>MDAwMENBNTggMDAwMkNDNkQgMDAwMDgyM0MgMDAwMDgwMEQgMDAwNDAxODIgMDAw +ND</line> ... </data>

You might imagine that by introducing all those extra <line> tags and the implicit extra structure, that the parsers would have more work to do and would therefore be slower. But here are the timings for processing the modified version:

Parser20000 lines
XML::Parser2
XML::SAX::Expat7
XML::SAX::ExpatXS5
XML::SAX::PurePerl73
XML::LibXML::SAX::Parser7

Yes, you read that right, the run time using XML::Parser dropped from 169 seconds to 2! This is a much more 'normal' result for a parser shootout. PurePerl is the clear loser and expat is the clear winner. LibXML is looking pretty good, but that parser's advantages don't really shine unless you're building DOM trees - when it moves into a class of its own.

So what's causing your problem? Dunno. You seem to have triggered some pathological corner case in expat. But really, it seems to me like a case of "Doctor it hurts when I do this". Had you considered passing just the filename for the MP3 file?

Update: Yeah thanks dbecoll - I obviously had my brain disengaged when I typed that.

Replies are listed 'Best First'.
Re^2: XML::Simple hangs
by dbecoll (Initiate) on Jun 05, 2005 at 07:01 UTC
    > As you can see, the run times for all the expat based
    > parsers are increasing exponentially with the size of the
    > file. The PurePerl parser is coping amazingly well and its 
    > run times are only increasing geometrically.
    
    I hope you meant linearly rather than geometrically. ;)
    
Re^2: XML::Simple hangs
by Ovid (Cardinal) on Jun 05, 2005 at 14:22 UTC

    Thank you very much. Regrettably, just passing the filename isn't an option in this case. This XML is being pass via SOAP and, as a result, the SOAP server will sometimes be remote and not always have access to the mp3. However, you've definitely given me some great information.

    Cheers,
    Ovid

    New address of my CGI Course.