If you're asking about perl this and above post on GD are relevant. Get the data out of Excel and reprocess it yourself. Then either make a new excel sheet with condensed data, or draw your own chart in GD.

I have had good results with parsing data from up to 1MB size spreadsheets (the parsing takes a little while at that size). I am plotting just a certain window in time though as bar charts with plain vanilla GD.

You could do lots of processing, but your eye is really good at picking things out too. So instead of figuring out how to reduce the amount of data I would like to suggest that you simply plot all of the data, perhaps with an appropriate transformation (stretch) to emphasize what you are looking for.

If it is just 1 and 0 well that's not so bad. But if it is fine differences then try and expand the spectrum of color you use too.. Once you have a finely drawn chart, explore it with a program that lets you change gamut and zoom in if you like, there may be a lot of detail you can't see easily but will become uncovered by changing the palette. For example xephem may show you what kinds of things are possible (I have the comet on my mind). It can find dim stars, and invert or add tails to make them easier to see. Or the gimp, though I have in an image processing program called NIH Image (or NIH ImageJ for windows/linux/osx). It's not just for biotech, and has an astronomy file as a sample which may be applicable to your problem. It has some interesting features although I don't think it lets you drag and adjust the lookup table (GLUT) in realtime anymore like photoshop does. Something that lets you do that and maybe zoom or view from different angles (if you have a 3d box for example) can help you find patterns extremely quickly.

Possibly an opengl model of it could be interesting too (I'm thinking of Wx::GLCanvas though search.cpan.org searching for "opengl" gave some likely suspects.) There's undoubtedly a lot of other stats programs out there that could be used, like R statistics language with the IDPmisc libary package for display of large datasets (adds effects to the display). And there is IBM's OpenDX (Open Visualization Data Explorer, which is open source). The Colormap Editor in the screenshot on that page is what I was talking about anyway it is a very cool system, only a 40 megabyte download away!

In reply to Re: intelligent re-sampling of data by mattr
in thread intelligent re-sampling of data by spurperl

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.