Would it be possible for you to store the data in a database table or some type of disk storage, rather than storing all of it in memory? That would make your program much more scalable...

That is not always possible, only in most cases. I once had to code something that had to parse over 200 MB of data, with absolutely no disk I/O - no ramdrives either. The box had 1 GB of ram, so that was not a problem.

Unfortunately, I had to re-invent a wheel. There already was a data parser for the particular format, but the author only thought of scalability on machines with harddisks.

When creating something that handles a lot of data, consider:

• RAM is faster than disk
• Disk usually has more free space
• Letting the OS swap to disk can be faster than handling I/O yourself (not always true)
• In-RAM filesystems can come in handy
• Writing large chunks is faster (generate > /tmpfs/foo; mv /tmpfs/foo foo can be faster than generate > foo
• Flush data BEFORE you run out of RAM.

My workstation has 1 GB of RAM, and another gigabyte of swap space. I tend to slurp files, do things in memory, and spit out entire files at once. This drastically decreases time spent on coding, and increases speed (most of the time). :)

U28geW91IGNhbiBhbGwgcm90MTMgY
W5kIHBhY2soKS4gQnV0IGRvIHlvdS
ByZWNvZ25pc2UgQmFzZTY0IHdoZW4
geW91IHNlZSBpdD8gIC0tIEp1ZXJk

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.