mohangopa:

I've found that hashes work even when the datasets are fairly large, especially if the hash entries themselves aren't too big. I'd go ahead and try it and see how well hashes work in your application.

Once you have it running, give it a file twice the normal size, and watch the memory it consumes as it runs. If all goes well, then I'd guess you're done until your processing volume doubles. If it had RAM trouble, then I'd suggest using something like SQLite or DBM to give you quick local cache with disk backup. That way, you can handle larger data structures without the overhead of communicating with the external database.

...roboticus

When your only tool is a hammer, all problems look like your thumb.