in reply to Re^2: XML::Twig and threads
in thread XML::Twig and threads [solved]

So you are right - 2 minutes is not much time. What worries me is 3.5GB RAM

I'll bet £1 to 1p that if you comment out the use threads;, the memory consumption will barely change.

It is not at all uncommon for a 100MB XML file to translate into 3.6GB of ram requirement once it has been parsed and the equivalent data structure constructed.

The memory requirement has nothing to do with threading. Just Perl's well-known tendency to trade memory for cpu.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong
blockquote

Replies are listed 'Best First'.
Re^4: XML::Twig and threads
by grizzley (Chaplain) on Nov 28, 2012 at 09:57 UTC

    I won't bet obviousness :)

    I've updated question node with example XML and script doing the job. Input file is read again in each of few hundred iterations. I was thinking about reading it once and then make in-memory copy in each iteration, but this will include swap on HDD and definitely won't speed up the code. Another way would be to store all changes write altered version to file, undo stored changes etc. That would save copying, but on the other hand as there are many changes - structure storing changes might be of same size as original. The best would be to somehow write to file original structure line after line and replace fragments on-fly.

[reply]

      Firstly, updating the root node with new code is a bad idea. Very few if any people will re-read old nodes, so it will never be seen except by those you notify.

      Second, posting code that doesn't use strict & warnings and contains trivial errors that those would catch:

      Use of uninitialized value $fields[2] in pattern match (m//) at C:\tes
+t\1005623.pl line 56.
Use of uninitialized value $fields[3] in pattern match (m//) at C:\tes
+t\1005623.pl line 64.

      [download]

      will stop most people (including me) from looking further.

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      RIP Neil Armstrong
[reply]
[d/l]
        Ok, noted. I just didn't want to start new discussion on the same topic.
[reply]

In Section Seekers of Perl Wisdom