PriNet has asked for the wisdom of the Perl Monks concerning the following question:

got a quick q... when you load an array, is the array size limited by the amount of ram in your machine? or does the interpeter cache the "extra" info to the hdd making the array size "unlimited" ?

I did try re-inventing the wheel...
But it kept getting stuck on the corners

Replies are listed 'Best First'.
Re: array size
by Anonymous Monk on Aug 10, 2012 at 00:48 UTC

    There is always a limit. Perl asks the operating system for memory. The operating system either gives perl however much memory perl asks for, or it gives it nothing. The operating system takes care of any "caching" to swap memory , but its not unlimited.

    See also how apply large memory with perl? (and the links therein)

[reply]
Re: array size
by Marshall (Canon) on Aug 10, 2012 at 16:51 UTC
    If you are running 32 bit Perl like myself on Windows, then 2 GB would be an absolute "theoretical" max size due to addressing limitations and you won't be able to get all of that for array data storage. My Win XP Pro system will "auger into the ground" in terms of performance well short of that (memory swapping, etc).

    Mileage varies a LOT and there are many variables involved. My largest memory hog takes about 300 MB+ of memory, but is compute and I/O bound - no significant memory swapping takes place - if that starts, you are "dead" - performance will plummet by orders of magnitude!

    You should realize that a Perl data structures have significant amounts of "bookkeeping".

    In practical terms, 100K things - absolutely no problem, 1 million also no problem, after that serious thinking should start about "why do I need multiple million things in memory at the same time?", especially a simple array structure which is hard to search.

    Time to think about other algorithms possibly including databases. I have become quite enamored with SQLite - this works extremely well with Perl. If you are thinking about running a forking server on Unix, then then you shouldn't be thinking about even getting close to a single process limit. Run "mean and clean."

    Summary: Memory usage still does matter. If you are on your own dedicated work station, that's one thing - if you want to run on a server, think even much more carefully about the algorithms and methods.

    If you could describe your problem that you figure requires "max memory", maybe the Monks can help you reformulate this. At this point, I have no idea. But there are very real practical limits.

[reply]

Back to Seekers of Perl Wisdom