Thanks! I'll give the snipped above a go.

Here is an example of what I'm doing. I'm downloading very detailed information on the wholesale price of electricity. In my market, there is an individual price for each of several thousand nodes in the market, and there is a separate price for each hour, and in the "real-time" market, for every five minutes.

The data comes in flat CSV files for each day that look like

node,hour,interval,price
bob,3,4,45.64
...
[download]

I frequently need to generate reports from this data such as:

• what is the price of energy at each hour averaged over all nodes (or weighted over nodes more heavily traded)?
• what is the price of energy at each node, averaged over all hours (or weighted over the hours more heavily traded)?
• what is the highest price observed in hour 16 over the last 5 weekdays?
• what day had the greatest hourly variation among the following n nodes in the past three months?
• various charts, graphs, histograms, maps, and statistical analyses, all generated across different "axes" of space and time.

In general, what I do is load up the data I'm interested in and plop it into a hash of hashes of hashes ... that is organized most conveniently for the burning management question du jour, but then the next day, a different question would be easier to calculate with a different arrangement. Often, these analyses get accreted into a daily report generated by one script, but I'm trying to minimize the amount of shuffling going on.

Run-time is an issue, but size is usually a bigger problem.

dave

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.