in reply to Tokenising a 10MB file trashes a 2GB machine

Be aware that using Devel::Size::total_size() itself can consume prodigous amounts of memory.

In the process of examining the structure of the anonymous array you are creating, it builds a 'tracking hash' to allow it to avoid counting duplicate references to data or internal magic etc. Whether that's the cause of your memory consumption here is not clear, but it would be worth eliminating Devel::Size from the equation.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

Replies are listed 'Best First'.
Re^2: Tokenising a 10MB file trashes a 2GB machine
by PetaMem (Priest) on Jul 16, 2008 at 09:23 UTC

    It should be obvious, that I added Devel::Size AFTER I found my script trashing the comp. I wanted to write in my original post that Devel::Size acts completely neutral in this case but found that information redundant.

    Bye
     PetaMem         All Perl:   MT, NLP, NLU

[reply]

In Section Seekers of Perl Wisdom