Re: How to Calculate Memory Needs?

Would it be possible for you to store the data in a database table or some type of disk storage, rather than storing all of it in memory? That would make your program much more scalable...

Otherwise, I'm surprised that I can't find some docs ANYWHERE that show how to calculate memory usage of data structures.

Update: Check out Camel 3, p. 504. Maybe PERL_DEBUG_MSTATS will help?

Update2: Hmm.. Who wants to play with some inline C and figure out a way to do a sizeof() to pull the memory usage of a hash? Is that possible?

Update3: Here's da goodz: http://take23.org/docs/guide/performance.xml/4. It has GTop examples. Fletch was all over that from the start. :)

Comment on Re: How to Calculate Memory Needs?

Replies are listed 'Best First'.
Re: Re: How to Calculate Memory Needs? by Juerd (Abbot) on Apr 05, 2002 at 16:27 UTC
Would it be possible for you to store the data in a database table or some type of disk storage, rather than storing all of it in memory? That would make your program much more scalable... That is not always possible, only in most cases. I once had to code something that had to parse over 200 MB of data, with absolutely no disk I/O - no ramdrives either. The box had 1 GB of ram, so that was not a problem. Unfortunately, I had to re-invent a wheel. There already was a data parser for the particular format, but the author only thought of scalability on machines with harddisks. When creating something that handles a lot of data, consider: RAM is faster than disk Disk usually has more free space Letting the OS swap to disk can be faster than handling I/O yourself (not always true) In-RAM filesystems can come in handy Writing large chunks is faster (`generate > /tmpfs/foo; mv /tmpfs/foo foo` can be faster than `generate > foo` Flush data BEFORE you run out of RAM. My workstation has 1 GB of RAM, and another gigabyte of swap space. I tend to slurp files, do things in memory, and spit out entire files at once. This drastically decreases time spent on coding, and increases speed (most of the time). :) U28geW91IGNhbiBhbGwgcm90MTMgY W5kIHBhY2soKS4gQnV0IGRvIHlvdS ByZWNvZ25pc2UgQmFzZTY0IHdoZW4 geW91IHNlZSBpdD8gIC0tIEp1ZXJk	[reply]
Re: Re: How to Calculate Memory Needs? by mrbbking (Hermit) on Apr 05, 2002 at 19:17 UTC
I figured I go with storing it on disk only if I can't do it in memory. I'm not sure right now exactly how I could use it from disk, but it works now if the whole thing is in memory. Here I must cite the virtue of laziness. (Though in this context, I'm not certain that it's still a virtue...) Found this in the Perl CD Bookshelf: *PERL_DEBUG_MSTATS Relevant only if Perl is compiled with the malloc function included with the Perl distribution (that is, if perl -V:d_mymalloc yields "define"). If set, this causes memory statistics to be displayed after execution. If set to an integer greater than one, also causes memory statistics to be displayed after compilation. ...but that gives me 'undef', so that's out for me. If your Inline C suggestion is possible, it's beyond my current capabilities... :-( So, I'm off to read all about GTop, then!	[reply]