building a time series form dated flat files

sean.snv has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks, I am trying to build a time series for a particular stock from data that exist in dated flat files for many stocks. I will try to illustrate a simple example, to be clear.

File1 - Data20100131.csv
stock1,10.2
stock2,9.5
stock3,34.7

File2 - Data20100228.csv
stock1,10.5
stock2,13.5
stock3,23.1

File3 - Data20100331.csv
stock1,11.1
stock2,9.2
stock3,44.6
[download]

Currently, I am specifying a date range and stock and get a time series. For example: timeseries("stock2", [20100228,20100331]) returns

{
  20100228 => 13.5,
  20100331 => 9.2,
}
[download]

The way I do this is to loop through each dated file line by line looking for my stock identifier, grabbing my data and moving on to the next file. Is there a more efficient/clever way of doing this? Thanks in advance, Sean

Comment on building a time series form dated flat files Select or Download Code

Replies are listed 'Best First'.
Re: building a time series form dated flat files by mr_mischief (Monsignor) on Oct 14, 2010 at 17:21 UTC
What's more efficient depends partly on what you do with the data. Are you going back through every file over and over again for different stocks many times? That's the impression I got. If so, putting the info in a database of some sort up front and querying the database will be much more efficient. If, though, you are only interested in a few stocks and only hit those files for a particular date range a few times, you might not want to bother with changing anything.	[reply]
Re^2: building a time series form dated flat files by sean.snv (Initiate) on Oct 14, 2010 at 17:40 UTC
Thanks for the response. That's what I wanted to know. If I was only hitting this once or twice, is there a better way to find the specific stock I want other than line by line until I get there? Otherwise, I will look to putting the data in a database if I need to access the data much more than that. Sean	[reply]