Looking for some design guidance

Aim9b has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: Looking for some design guidance by BrowserUk (Patriarch) on Aug 17, 2007 at 12:24 UTC
Do you need totals per column per file, or totals per column for all files? Ditto, are the per row percentages: The sum of this row is x% of the total of all values in this file? Or for all files? Or one percentage for each column of that value relative to the column total for this file? Or all files? Or one percentage for each of the five groups of 4 values relative to this file? Or all files? Your 'spec' leaves so much unclear. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply]
Re: Looking for some design guidance by moritz (Cardinal) on Aug 17, 2007 at 12:16 UTC
If you don't want the plain Array of Arrays-approach, you could build a Spreadsheet::Perl in memory. Or if you want to stick to the database metapher, you could use DBI on top of DBD::AnyData or DBD::CSV. Your data specs sounds as if the data would never exceed a few Megabytes in size, I woudn't worry about not keeping it in memory at all. Perl 6 in German	[reply]
Re: Looking for some design guidance by McDarren (Abbot) on Aug 17, 2007 at 12:30 UTC
Depending on the exact nature of your data, you're probably looking at a HoH (Hash of Hashes) or a HoL (Hash of Lists). The PDSC gives a good introduction and overview of these. As far as storing the data, apart from a database there are several other options, including a plain old text file. Storable might be a quick and easy solution. It would be helpful if you could post a half-dozen lines of sample data from your CSV file. Cheers, Darren :)	[reply]
Re: Looking for some design guidance by BrowserUk (Patriarch) on Aug 17, 2007 at 16:20 UTC
Sorry, but isn't "Access" a database? (Runs away and does a google. Comes back with a quizical look on face) You already have the data in a database, with all the power of SQL at your finger tips and you want to resort to exporting the data to a csv file and importing it into Perl, in order to do some SQL-bread&butter totalling and percentage calculations? There surely must be someone local to you that knows enough SQL to do this for you. It's probably one, at most two sql statements. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply]
Re^2: Looking for some design guidance by Aim9b (Monk) on Aug 18, 2007 at 00:58 UTC
I'm sorry for the confusion...no they are in cvs files now. They get loaded into an access db for processing. In my first of my 3 less-than-stellar posts I had a typo that said 'is working fine', but really it should have been 'isN'T' working fine. The import functions ok unless you try to import the 1st or last file, & when the year rolls over, there is a 2 day waiting period. I could probably fix it in VBA, but it seemed like a great chance to exercise & expand my perl. Maybe I'm wrong. I guess time will tell. Also, every now & then the thing crashes & I have to manually delete several tables just to get it to run again. It's a single user app & he's very patient. It's been thru 3 Windows/Office upgrades. I'd just hoped to enhance it for him in a more stable environment. I'm doing this for free outside of work, just because it saves him about 14 hours that he puts into Excel on his weekend & I need the learning. Thanks again for the help. The monestary has been immeasurably helpful.	[reply]
Re^3: Looking for some design guidance by BrowserUk (Patriarch) on Aug 18, 2007 at 01:33 UTC
The data looks like this, representing customer counts & sales in 15 min incr. I need totals & percentages for each day, by section (cols 3,4,5/6,7,8, etc). `12/17/2002,0700,001,0000,00000.00,000,0004,00022.15,... 12/17/2002,0715,001,0000,00000.00,000,0012,00036.38,... 12/17/2002,0730,001,0000,00000.00,000,0014,00048.18,... 12/17/2002,0745,001,0000,00000.00,000,0016,00142.67,... 12/17/2002,0800,001,0000,00000.00,000,0023,00008.28,... 12/17/2002,0815,001,0000,00000.00,000,0052,00419.79,... 12/17/2002,0830,001,0000,00000.00,000,0047,00026.52,... 12/17/2002,0845,001,0000,00000.00,000,0071,00028.81,...` [download] Okay. But your still holding back on the details. And there are anomolies in what you are saying. You say the figures represent "customer counts & sales", but then say a section consists of 3 columns? And when you originally posted, prior to wiping it out with your first update, you said 5 sets of 4 columns? You say the figures are in 15 minute increments, but originally said each file contained 80 rows. The web site closes for 4 hours each night? You say "I need totals & percentages for each day, by section". Totals down each column? Totals across each row? What percentages? How many percentages? You also originally mentioned 366 files, and now mention year end rollover, but make no mention if you are trying to roll up the daily stats across the year? Finally, you are "doing this for free", to help someone out--and then come here and ask for help. You offer no code. And give no clues as to what part you are having a problem with? Do you know how to open and read a file in Perl? I'll let you think about what implications people might draw from that lot. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l]
Re^4: Looking for some design guidance by Aim9b (Monk) on Aug 20, 2007 at 12:52 UTC