neversaint has asked for the wisdom of the Perl Monks concerning the following question:

Dear Masters,
I have a large code (mycode.pl), it takes a single file as input and return some results. Now, most of the time I will encounter this warning in my code, upon returning the result my code is terminated as well.
Out of memory!
Attempt to free temp prematurely: SV 0xe70a608, 
Perl interpreter: 0x8b1a008, <INFILE> line 24.

Attempt to free unreferenced scalar: SV 0xe70a608, 
Perl interpreter: 0x8b1a008, <INFILE> line 24.

[download]
My questions are:

---
neversaint and everlastingly indebted.......

Replies are listed 'Best First'.
Re: Attempt to free temp prematurely and unreferenced scalar
by BrowserUk (Patriarch) on Feb 22, 2006 at 04:29 UTC

    The significant part of that error message is the "Out of memory!", the other part is probably just a side-effect of that initial error.

    Your applicaton is using more memory than you have available. Given the reference to <INFILE> it seems likely that you are attempting to read a very large file, bigger than your combined physical and virtual memory into RAM. In most cases, processing large files is done sequentially and there is little benefit in loading the whole thing into RAM.

    For those applications that require random access to the data, it's convenient to do so, but once you reach the point that you exhaust physical RAM, you loose most of the performance benefits through swapping anyway. At that point, creating a ram-based index to the lines of data and only loading the actual lines when required can be extremely effective alternative for seldom accessed or constantly changing flat file data.

    For frequently accessed and/or infrequently changing data*, putting it into some form of database is the way to go.

    *By this I mean where the bulk of the data doesn't change frequently.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
      At that point, creating a ram-based index to the lines of data and only loading the actual lines when required can be extremely effective
      Dear BrowserUk,

      Would you mind give us the simple example of the above approach?
      I can't seem to figure out how to do that in Perl.

      ---
      neversaint and everlastingly indebted.......
[reply]

        Are you after read-only or read-write acess to the file in question?

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]
Re: Attempt to free temp prematurely and unreferenced scalar
by GrandFather (Saint) on Feb 22, 2006 at 01:18 UTC

    Put use diagnostics; in your code. You should then get a call stack when things go pear shaped.

    DWIM is Perl's answer to Gödel
[reply]
[d/l]
      Dear GrandFather,
      I've tried your suggestion, still it returns the similar massage as my OP into STDERR.

      ---
      neversaint and everlastingly indebted.......
[reply]
Re: Attempt to free temp prematurely and unreferenced scalar
by ryanc (Monk) on Feb 23, 2006 at 16:47 UTC
    how are you loading and moving through your file?

    something like
      while (<INFILE>)
    will attempt to load the entire file into memory, while:
      while (defined(<INFILE>))
    will just read one line at a time.

    ryanc

[reply]
      No they are the SAME. Because the compiler adds the "defined" when it sees a diamond only in a "while".
[reply]

Back to Seekers of Perl Wisdom