in reply to Attempt to free temp prematurely and unreferenced scalar

The significant part of that error message is the "Out of memory!", the other part is probably just a side-effect of that initial error.

Your applicaton is using more memory than you have available. Given the reference to <INFILE> it seems likely that you are attempting to read a very large file, bigger than your combined physical and virtual memory into RAM. In most cases, processing large files is done sequentially and there is little benefit in loading the whole thing into RAM.

For those applications that require random access to the data, it's convenient to do so, but once you reach the point that you exhaust physical RAM, you loose most of the performance benefits through swapping anyway. At that point, creating a ram-based index to the lines of data and only loading the actual lines when required can be extremely effective alternative for seldom accessed or constantly changing flat file data.

For frequently accessed and/or infrequently changing data*, putting it into some form of database is the way to go.

*By this I mean where the bulk of the data doesn't change frequently.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^2: Attempt to free temp prematurely and unreferenced scalar
by neversaint (Deacon) on Feb 22, 2006 at 10:14 UTC
    At that point, creating a ram-based index to the lines of data and only loading the actual lines when required can be extremely effective
    Dear BrowserUk,

    Would you mind give us the simple example of the above approach?
    I can't seem to figure out how to do that in Perl.

    ---
    neversaint and everlastingly indebted.......

      Are you after read-only or read-write acess to the file in question?


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        Dear BrowserUk,

        After Read-only. I've also just looked through the input file size. I realized that it's still within reasonable size (5k-60k), where I have 1GB RAM.

        I've tried to see the size of the some potentially large variable using Devel::Size. All of them are still reasonable (around 20MB).

        Apart from my question to your posting. Is there a way to check, which part of my code are producing the "Out Of Memory" message? like stated in my OP.


        ---
        neversaint and everlastingly indebted.......