Re^2: Research App Optimization

Hi Tilly, Thanks for your response. Yes we are looking at atleast 20 to 50 thousand responses, answering around 50-80 questions. Also the same respondents could be asked more questions in the future, so the question database would go on increasing over time. I looked at Data::Mining module, but it doesn't cater to what I need. I am looking at a perl hash of hash right now and storing them into a file using Storage.pm or something and loading it in memory when required, to reduce the time. There could be soooo many associations and rules, I am thinking of writing a rule-engine too, I researched on yagg yesterday. It could be like In year 1980 in city 'new york' maxSold fabrickType for age-group 20-25 ? Can I store this historical data in the db? or I am better off using hashes. I preferred XML, as my hash would easily form an XML DOM Tree.

      <results>
         <result id='1'>
             <year = '1980'/>
                <sales>
                   <sale>
                     <location>New York</location>
                      <cotton>5.6</cotton>
                      <nylon>8.4</nylon>
                     </saled>
                </sales>
         </result>
      </results>
[download]

I will look into PDL and Statistic Package as well. Thanks All. Sandeep

Comment on Re^2: Research App Optimization Download Code

Replies are listed 'Best First'.
Re^3: Research App Optimization by tilly (Archbishop) on Nov 01, 2006 at 23:19 UTC
Databases are for data. With that little volume, there will be no problem storing it all in the database and worry about what you're going to do with it later. Yes, including historical data. And for intermediate manipulations, the database already can do a lot of what you want - you just have to write the query. Should you change your mind, it is easy enough to export it all into any format you want. And if you're worried about performance, don't. Databases are a lot faster than searching through and processing XML files. (In fact processing XML is a very silly thing to do if you care about performance.) However one note. You'll want to think through what information you're capturing, and how to organize it. Databases really start to shine when you structure your data appropriately for them. In fact if you can, I'd suggest finding someone local who knows databases, discuss the problem with them, and have them suggest a table structure for you.	[reply]

Replies are listed 'Best First'.

Re^3: Research App Optimization
by tilly (Archbishop) on Nov 01, 2006 at 23:19 UTC

Should you change your mind, it is easy enough to export it all into any format you want. And if you're worried about performance, don't. Databases are a lot faster than searching through and processing XML files. (In fact processing XML is a very silly thing to do if you care about performance.)

However one note. You'll want to think through what information you're capturing, and how to organize it. Databases really start to shine when you structure your data appropriately for them. In fact if you can, I'd suggest finding someone local who knows databases, discuss the problem with them, and have them suggest a table structure for you.

[reply]