Re: Libxml parser cosuming 100% cpu

If you could provide something we can actually run to see the performance issues ourselves, we could provide specific suggestions - see Short, Self-Contained, Correct Example. Until then, some general optimization tips:

Use Devel::NYTProf to locate the places where your code is spending the most time.
Consider using a different XML parser, since XML::LibXML will load the entire DOM structure into memory by default, while XML::Twig is geared more towards processing a large file piece by piece, without loading the whole file into memory.
I see you doing foreach over an array of apparently 1 million (?) lines - this means you're keeping those lines in memory. It would most likely be more efficient to use a while(<$filehandle>) loop to read the file line-by-line, without loading it all into memory.
I see you repeating some findnodes calls twice, causing that effort to be doubled. Consider storing the results in a local variable so you can use it more than once.

If you have a input file of 1 million lines, consider that maybe it'll just take some time to process, and that 100% CPU consumption during that time is normal.

Comment on Re: Libxml parser cosuming 100% cpu Select or Download Code

Replies are listed 'Best First'.
Re^2: Libxml parser cosuming 100% cpu by geek2882 (Initiate) on Aug 11, 2018 at 11:43 UTC
Thanks, (first point) But i think if i use XML::Twig then execution time will be more than 2.5min. second thing i got it XML::LibXML store the doc in memory but its a small record of 100 Lines around ? do you think XML::Twig reduce the cpu usage and fast the execution. (third point) Which findnodes you are talking about. `i have some more question here Which one is fast there in each two opt +ion: 1). $method = $xml->getChildrenByTagName('Method')->to_literal; $method = $xml->fidnnodes('./Method')->to_literal; 2). $xml->findnodes('./Indi/Lost/true') $xml->exists('./Indi/Lost/true')` [download]	[reply] [d/l]
Re^3: Libxml parser cosuming 100% cpu by haukex (Archbishop) on Aug 11, 2018 at 13:37 UTC
But i think if i use XML::Twig then execution time will be more than 2.5min. When it comes to optimizations, I'd suggest not "guessing" which might be faster, but measuring and testing! i got it XML::LibXML store the doc in memory but its a small record of 100 Lines around ? do you think XML::Twig reduce the cpu usage and fast the execution. If I understand correctly that you're splitting your ~1 million line file into chunks of 100 lines and then processing those one at a time, then I would agree with the guess that XML::Twig might not give you a big speed boost (unless you have ridiculously long lines). On the other hand, if your ~1 million line file is one big, well-formed XML file, then you would be able to get rid of your custom "splitting" code and use XML::Twig to process the entire file, one "record" at a time. If you need help with that, you'd have to show us some sample input (see How do I post a question effectively? and Short, Self-Contained, Correct Example). Which findnodes you are talking about. I was talking about this: `if ($xml->findnodes('./Indi/Lost/true') ) { } if ( $xml->findnodes('./Indi/Lost/true') \|\| $xml->findnodes('./Indi/Lo +sgshshsht/false') ) { }` [download] Which is better written like this, to avoid the doubling of the `findnodes` call (Update: unless of course the first `if` block makes modifications to the document that would require the second `if` to re-run the `findnodes`): `my $result = $xml->findnodes('./Indi/Lost/true'); if ( $result ) { } if ( $result \|\| $xml->findnodes('./Indi/Losgshshsht/false') ) { }` [download] And a similar thing with `$xml->findnodes('./nike'))[0]->firstChild`. Which one is fast there in each two option I don't have the time to test right now, but the go-to module for this kind of comparison is Benchmark. But as I said before, measure where your code is spending the most time with Devel::NYTProf, and then optimize those places, instead of guessing and doing what might turn out to be an unnecessary micro-optimization.	[reply] [d/l] [select]
Re^3: Libxml parser cosuming 100% cpu by ikegami (Patriarch) on Aug 11, 2018 at 12:49 UTC
If your saying your XML doc has only ~100 nodes, then the problem isn't memory, then using a pull parser like XML::Twig (or XML::LibXML::Reader) won't help.	[reply]