in reply to Reading huge file content

Despite comments to the contrary above, available RAM is not a factor in how much stuff you can read into "memory". Virtual memory means that the physical RAM available is a performance limiting factor rather than an absolute limit. However, there are absolute limits and, depending on OS and build of Perl, 4 GB is likely to be beyond the maximum chunk of (virtual) memory you can use.

The answer, as suggested elsewhere, is to restructure your code so that the analysis sub uses either a file handle and reads the data directly, or uses a database, or use something like Tie::File to avoid loading all of the file into "memory".

Perl is environmentally friendly - it saves trees

Replies are listed 'Best First'.
Re^2: Reading huge file content
by KurtSchwind (Chaplain) on Dec 06, 2007 at 22:49 UTC

    Are you sure?

    I'm fairly certain that your physical RAM limits how much you can read into a single memory segment. That is, if you want to read everything into a single variable, you'll need the physical RAM to do it. Virtual memory can only swap out what you aren't currently accessing.

    Am I wrong on this?

    --
    I used to drive a Heisenbergmobile, but every time I looked at the speedometer, I got lost.
[reply]

      Yes, you are wrong. ;)

      Approximately: virtual memory systems give a process a chunk of memory that looks to the process like one large contiguous piece of memory. The process can not tell where or how its process space memory is mapped to physical memory. Indeed that mapping may change from access to access. The OS takes care of ensuring that when a process accesses a chunk of process space memory that the access succeeds if possible. That may entail writing a chunk of physical memory to disk, reading another chunk of disk to get the accessed process memory's contents, then fixing up tables to map process memory address to physical addresses. Hitting on the VM doesn't come cheap!

      Perl is environmentally friendly - it saves trees
[reply]

        I'd check again.

        Virtual memory will allow your processes to allocate more memory than you have physical ram, provided that you do not need to process it all as a single contiguous chunk.

        But the largest single contiguous chunk (eg. string) that you will be able to allocate will always be somewhat less than your total physical ram.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom