Hello all, I have a written a "chart parser" for Math notation, and I have written it entirely in Perl.

It works by building tree structures corresponding to every valid substring of the input, and storing them in memory. Then later on, it may combine some of the subtrees to make larger trees, which also need to be stored.

The problem is that a malicious user (or just a very ignorant one) can enter inputs that are so ambiguous that they will cause a great many trees to be built during the course of the parse.

It is possible to run the parser out of memory this way, and indeed it has already happened.

Another thing that makes this a difficult problem is that the "Out of Memory" error can occur at different places in the code. Sometimes when it tries to construct a new "Node" object. Sometimes, if it's already running low on memory, the error will happen when it's just adding an element to a pre-existing hash.

My question is, "Is there a way to monitor the free memory available from within the program?"

I've looked a little bit at Devel::Peek, and it seems promising, but I figured I'd check with you guys first. Surely someone else has run across this same issue!

Barring that, I'm thinking I may have to use the C programming language to rewrite my own versions of each basic data structure (array, hash, node), as XSUBs. Then they each could have a "safety valve" which would use malloc to check and see if there's actually enough memory to create or extend the given data. If not, the function could return an error value which would make the parsing stop, and send a message to the user.

One thing I've already tried is simply putting a limit on the amount of "Node" objects that can be created during parsing, but this is not a real solution. Apparently the amount of free memory for the process can vary depending on various factors.

Any help will be greatly appreciated!

~tford

In reply to Prevent "Out of Memory" error by tford

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.