Framework for handling vast reams of tabular data.

Socrates has asked for the wisdom of the Perl Monks concerning the following question:

I'm working on a project that involves network traffic analysis. This, of course, involves lots of data on individual network events or groups of related events.

What I seek is a (Perl) framework that can help me encapsulate them and display them in a variety of fashions (e.g. chronologically, by priority, globbed by "type"), and allow for sorting, constraining and such.

I researched using a MVC (like Catalyst), but they all seem to be geared towards handling database calls and representation transparently for someone starting from scratch. In other words, they assume you will be describing to them the type of data you'll be storing and that the MVC will handle the particulars.

This is a problem because I am already tied to a particular database representation and to the libraries which access this data for me.

To be sure, these libraries will, in large part, guide the data structures/objects I design, but it would be nice to be able to plug these objects into a framework that can help me deal with displaying, sorting, and grouping them.

Any guidance?

UPDATE:

I should have been more clear. It's not that I'm evaluating whether or not to use Perl. The project is actually fairly tied to Perl already (which is fine by me, I love Perl). What I meant by saying that I was tied to a particular representation and library is that there is already a system in place to display this data; I've been tasked with replacing it. So, I already have a database schema. Also a new system of libraries for accessing this database is already being written by another developer, so, while I have input into the process, it's not mine. So, what I'm asking about is whether there is a flexible Perl framework for displaying and sorting objects based on certain rules, that can also handle displaying them. I need to be more clear on that as well: this is for a web application (or one accessed in a browser), so tt would be nice to have a system of classes set up to accept objects and render them into a tabular display. Automatic generation of sorting routines and grouping routines would be nice, too. Basically like how you can bind data sources to display controls in .NET.

Comment on Framework for handling vast reams of tabular data.

Replies are listed 'Best First'.
Re: Framework for handling vast reams of tabular data. by Corion (Patriarch) on Jun 11, 2008 at 15:55 UTC
I'm not sure what problem you're trying to solve. I'm using Querylet to create reports from SQL queries. You mention a "particular database representation" and "libraries which access this data", but don't tell us which ones they are, so I'm assuming that you have a relational database with the data organized as tables, and the DBI module to access them.	[reply]
Re: Framework for handling vast reams of tabular data. by jettero (Monsignor) on Jun 11, 2008 at 15:53 UTC
I think, personally, I'd use perl itself (if possible). You can use sort, grep, map, and split to do almost everything you'll need. -Paul	[reply]
Re: Framework for handling vast reams of tabular data. by apl (Monsignor) on Jun 11, 2008 at 16:05 UTC
Perl is the solution for you. Curtesy of Data Type: Hash, you can store serially retrieved data in a variety of representations, later sorting each representation as you wish.	[reply]
Re: Framework for handling vast reams of tabular data. by pc88mxer (Vicar) on Jun 11, 2008 at 17:37 UTC
I think your approach will be influenced by the degree of user interactivity you want. If you want to be able to interactively rearrange, resize, hide and sort columns, etc., you'll want to do a lot of the presentation and user interaction with Javascript. There is a multitude of JS libraries available to display and manipulate tabular data. Some that come to mind are: Yahoo's YUI and ExtJS.	[reply]
Re: Framework for handling vast reams of tabular data. by Socrates (Acolyte) on Jun 11, 2008 at 16:19 UTC
I should have been more clear. See edit and revisions above.	[reply]