Would it be possible for you to store the data in a database table or some type of disk storage, rather than storing all of it in memory? That would make your program much more scalable...

That is not always possible, only in most cases. I once had to code something that had to parse over 200 MB of data, with absolutely no disk I/O - no ramdrives either. The box had 1 GB of ram, so that was not a problem.

Unfortunately, I had to re-invent a wheel. There already was a data parser for the particular format, but the author only thought of scalability on machines with harddisks.

When creating something that handles a lot of data, consider:

• RAM is faster than disk
• Disk usually has more free space
• Letting the OS swap to disk can be faster than handling I/O yourself (not always true)
• In-RAM filesystems can come in handy
• Writing large chunks is faster (generate > /tmpfs/foo; mv /tmpfs/foo foo can be faster than generate > foo
• Flush data BEFORE you run out of RAM.

My workstation has 1 GB of RAM, and another gigabyte of swap space. I tend to slurp files, do things in memory, and spit out entire files at once. This drastically decreases time spent on coding, and increases speed (most of the time). :)

U28geW91IGNhbiBhbGwgcm90MTMgY
W5kIHBhY2soKS4gQnV0IGRvIHlvdS
ByZWNvZ25pc2UgQmFzZTY0IHdoZW4
geW91IHNlZSBpdD8gIC0tIEp1ZXJk