Re^2: Perl Caching

I am really sorry guys. I did not state the problem correctly. Typically we update the securities once a day and our transaction feeds run through out the day. After the process runs that updates security master, I am building a cache file (huge -- close to couple of million). I want to use this cache file to store the asset_id as a new column in my transaction feeds, so I can avoid caching (lookup) while loading the warehouse. I can work on a flag to see whether my cache file is ready to be used or not. Should I be using hash to look up cusip/sedol/isin and pick up asset_id and how does this perform with large data files? Thanks and appreciate your response. Cheers. Raj

Comment on Re^2: Perl Caching

Replies are listed 'Best First'.
Re^3: Perl Caching by roboticus (Chancellor) on Aug 08, 2013 at 17:05 UTC
mohangopa: I've found that hashes work even when the datasets are fairly large, especially if the hash entries themselves aren't too big. I'd go ahead and try it and see how well hashes work in your application. Once you have it running, give it a file twice the normal size, and watch the memory it consumes as it runs. If all goes well, then I'd guess you're done until your processing volume doubles. If it had RAM trouble, then I'd suggest using something like SQLite or DBM to give you quick local cache with disk backup. That way, you can handle larger data structures without the overhead of communicating with the external database. ...roboticus When your only tool is a hammer, all problems look like your thumb.	[reply]

Replies are listed 'Best First'.

Re^3: Perl Caching
by roboticus (Chancellor) on Aug 08, 2013 at 17:05 UTC

mohangopa:

I've found that hashes work even when the datasets are fairly large, especially if the hash entries themselves aren't too big. I'd go ahead and try it and see how well hashes work in your application.

Once you have it running, give it a file twice the normal size, and watch the memory it consumes as it runs. If all goes well, then I'd guess you're done until your processing volume doubles. If it had RAM trouble, then I'd suggest using something like SQLite or DBM to give you quick local cache with disk backup. That way, you can handle larger data structures without the overhead of communicating with the external database.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

[reply]