I'm going to go against the grain a bit here... I don't believe an RDBMS would be right for you. You don't seem to need flexibility. You just need speed. Moving to an RDBMS now is also likely to be costly in terms of code changes.

I do think that you should be indexing your "shell" data though. I did something similar years ago by using an in-memory BerkeleyDB (DB_HASH). (It just so happens that I was working for a financial management company too.) Although what I did was all in C, I think you could actually do this pretty easily using BerkeleyDB from Perl and get all the real benefits. Keep in mind that, even though a 700MB file really isn't a big deal when you've got 16GB of RAM, Perl's representation of that data is going to include a lot of overhead, so it's best if you can skip it.

Doing it that way should have a pretty minimal affect on your code, all things considered. In fact, you might find it simplifies things. You won't need any heuristics to determine how to search your "shell"... you'll just iterate through your holdings doing lookups. I think that boils down to O(N) vs. O(N*M) in your current situation.

That's the direction I'd be looking anyway.

-sauoq
"My two cents aren't worth a dime.";