Re^2: More than one way to skin an architecture

Frankly, that question is the very reason I asked this question. Even though the dataset might not get very large, and therefore I could probably afford a "sloppy" approach to this problem, it seems prudent to really think this one through. Who knows? I might decide to keep a year's worth and then where would I be?

I running this on my "work" laptop. It's got all the usual Office apps running - Outlook, Word, PPT, etc., plus Firefox (which seems to be a memory hog, if you ask me). This planet program is grabbing the USGS every 20 minutes, airplane data every 5 (if I turn that on) and updating the day/night terminator line every 5. Since this script is going to run once a day (well, ships actually update about 4x/day but I don't know if I care that much), I don't want to drag this thing down just because I was sloppy about hunting down cruise ships.

The resources for this problem are finite; I don't have all the silicon I would like (I wish I did!); I work in an enterprise software company and bad architecture just offends me.

I didn't know that large hashes can cause a memory problem. It would be great to hear more about that kind of thing. And your comment,

"The key issue here is to be aware of what your chosen implementation is going to do, and how it's going to behave on your hardware"

is spot-on; except that I don't know what a given implementation might do - I'm new to Perl and therefore the question.

Comment on Re^2: More than one way to skin an architecture

Replies are listed 'Best First'.
Re^3: More than one way to skin an architecture by bigearsbilly (Novice) on Mar 19, 2008 at 11:36 UTC
you could always use a DBM file (AnyDBM) standard modules. they behave just like a hash, are dead easy to use and fast. They are simple key->value databases stored on disk. I used one with 250,000 price references once and was dead fast. hash version was very slow to load. updating with new CSVs would be easy too.	[reply]