|
Re: Variable stack size in Perl.
by BrowserUk (Patriarch) on Nov 11, 2002 at 12:51 UTC
|
The simple answer is that the size of memory available to Perl (under NT/AS at least) is (roughly)
(memory size on the server you are running on) + (the size of the swapper defined)
- (amount of memory that is used by perl itself + memory used by other programs on the server)
However, if the size of the file is a concern (and if this is a hosted domain especially), then you should think about accessing it contents in smaller chunks rather than all at once. There are many ways you can do this, but deciding or advising which one is right for your application will depend on knowing a lot more about what you are doing, and how you are doing it.
You might take a look at the various Tie::* modules that come as a part of your standard distribution and/or the various flavours of DBI on CPAN.
Nah! You're thinking of Simon Templar, originally played (on UKTV) by Roger Moore and later by Ian Ogilvy
| [reply] |
|
|
I want really to thank you, UK, j'm remained surprised: you're been quick and complete. But j'd like to understand better this think: would it tell that if the server is busied by other weavy processes, could it succeed that the reading of even a small file could cause Perl to raise an "out of memory" error? Always, if you know, is there an amount of memory that is affordably available, guaranted by ActivePerl?
| [reply] |
|
|
Essentially, no. There is no minimum amount of memory guarenteed to be available to Perl.
The restriction, if there is one, is the amount of memory available to the OS itself. If you are getting "out of memory" when trying to load a small file into memory, this is an OS configuration problem, not a problem with Perl itself.
It means that either the server has too little memory installed for the use to which it is put. Or,
Not enough room has been allocated to the swap space. Or,
That the servers discs are so full that it cannot expand it swap space to accomodate even small growth. Check the free space on the drive(s) on which the swapper has space allocated.
If either of these latter two is the cause, it is almost certain that the first is true as well or that the box has simply too much running on it.
Nah! You're thinking of Simon Templar, originally played (on UKTV) by Roger Moore and later by Ian Ogilvy
| [reply]
[d/l] |
|
|
| [reply] |
Re: Variable stack size in Perl.
by fglock (Vicar) on Nov 11, 2002 at 12:54 UTC
|
Perl memory size is only limited by your amount of
RAM and virtual memory. If it exceeds RAM size
the program will run slower, because it
will do disk accesses. If memory usage
grows even more, it will eventually give
an 'out of memory' error and stop.
| [reply] |
|
|
On an interesting additive side note, Perl also allows for the use of last-ditch memory should that last case ever happen. From the Camel's mouth: "However, if your perl was compiled to take advantage of $^M, you may use it as an emergency memory pool. If your Perl is compiled with -DPERL_EMERGENCY_SBRK and uses Perl's malloc then...", etc. It isn't used much, and probably shouldn't be (especially by a novice user)..but it's there.
| [reply] |
Re: Variable stack size in Perl.
by gjb (Vicar) on Nov 11, 2002 at 13:38 UTC
|
You might want to have a look at Tie::File on CPAN, this might do what you want.
(I know this was suggested by others, but looking at Tie::* returns an awful lot of results, and this one turns up at rank 100, so I thought it might be useful to be more specific.)
Hope this helps, -gjb-
| [reply] |
Re: Variable stack size in Perl.
by Elian (Parson) on Nov 11, 2002 at 15:56 UTC
|
You're generally limited by the amount of memory you have available to your process, rather than the stack size, as perl's variables aren't allocated on the stack. There are a couple of caveats, though.
- Arrays require a chunk of contiguous memory that's no smaller than (array_size * pointer_size) in bytes. If your array is 100 elements long and have 4 byte pointers, you'll need at least 400 bytes contiguous.
- When extending an array, perl may temporarily need a contiguous chunk of memory that's 1.2 times larger than the current size of the old array's contiguous memory needs. So if you extended the array in the previous point, perl would need a chunk of memory that's 480 bytes.
- Each string need to be in contiguous memory, so a 10K string would need a contiguous chunk of memory 10K in size
- Perl uses rather more memory than you might think, so it's easy to chew up a lot, just in absolute terms
The contiguous memory needs are often what gets you. If perl needs to extend a 50M element array it'll need temporarily another 240M of contiguous memory. Your process may have that much free in absolute terms, but not a single piece that big. (And some OSes and C libraries limit the size of a single memory allocation)
There are some tricks you can use, for example preextending and then shrinking an array, but if you need that you're dancing at the edge of disaster, so you might want to consider other solutions. (Which is why I'm not listing them. If you figure them out, you'll figure out when to use them as well, which is more useful, and much more difficult to explain)
You might also want to look at Devel::Size as something to work with in development. It'll give you an idea of how much memory your variables are using, which can help in planning coding strategies. | [reply] |