The PDL::Graphics engine may also have some way to display that type of data as colored flux density plots.

BrowserUK,
I may not be understanding this correctly but why not have the cone itself represent change in frequencies (or an illustration of overlapping cones) with HSV dots/circles positioned according to it's width(?). This is one of those cases where it's easier and faster for me to draw what I'm thinking of so here goes:

Note: SVG output using Illustrator so please forgive the bloated markup :)

If this isn't quite it, let me know, I have some time this afternoon for a quick edit or two. Hope this helps.

Luis, many thankyous for your time. That is certainly a pretty picture :)

I'm not yet sure that I understand what it displays, but I am very interested in how you produced it? What is Illustrator?

It took me a while to work out that I could view it directly in my browser. (Aside: Would be nice if we could embed such things in our PM posts -- though I can feel the resistance to that suggestion building even before I hit 'create' :)

What might make my understanding of the representation easier is if you could also post the basic data points that the image represents?

Sorry about not clarifying how to view the image. I used Adobe Illustrator CS5 but you could get away with Inkscape or maybe even using Cairo then exporting an SVG file.

I just read your longer reply and would like to take a day to think about it if that's alright. Let's ignore the sketch I made for now. :-)

"...the adversities born of well-placed thoughts should be considered mercies rather than misfortunes." — Don Quixote

I suppose better late than never. You've probably solved this and moved on but I increasingly feel a good candidate to represent a large volume of data like this would be a heatmap. I'm pretty certain that through color value and saturation you could abstractly accommodate data volume and frequency.

In addition to the heatmap examples (starting with the fifth one down) on the following page there are number of well executed 2 dimensional graphs.
http://cs.jhu.edu/~razvanm/fs-expedition/tux3.html



"...the adversities born of well-placed thoughts should be considered mercies rather than misfortunes." — Don Quixote

You could make a 3D version of a bubble chart. Using a good graphics library like OpenGL, your bubbles could really look like bubbles, with transparency and reflection and whatnot. That would be cool.

Okay. I guess I should have supplied some data and some explanation of what I've tried to date. (I didn't want to influence responses or cloud the underlying question with specifics of how and with what tools to do the actual drawing.)

Here is an illustrative, though rather small set of sample data. The keys are HSV (all scaled 0 .. 1 rather than H being 0 .. 360 ). The values are the frequencies of pixels with that HSV value within the image being analysed:

And here is a 2D/3D representation of that dataset. The H,S,V -> X,Y,Z in a 256x256x256 'cube'. The color of the plotted points is the pixel color.

As you can see, the HSV space discriminates the points into two very clear clusters when viewed this way. However, if viewed from different angles, it is possible to discern 5 or even eight clusterings with a few outliers.

If you look at the frequency data, it is possible to also find two very obvious peaks; and 4 or 5 or 7 more, depending where you apply the cutoff. The challenge is to try and a) find that cut-off; b) partition the dataset around those peaks.

I've made no attempt in this image to plot the frequency. The problem with trying to represent the frequency is the range of the values -- from 1 up to 8.7 million. It could be much bigger for larger images.

• Using color: No color ramp can cover this kind of range.

  Even if I used logarithms of the actual frequencies, the human brain is not well adapted to intuiting peaks and troughs of this magnitude from single points of color, let alone if they are logarithmic. Color ramps work well for surfaces where you can visualise darker colors as 'low' and brighter colors as 'high', because of the flow (gradation) between them.

  Not so good for this though.

• Using the size of the pixel/circle/voxel drawn within the 3D space.
  • Problem 1. Obliteration.

    If I draw the 'point/circle/volume' directly proportional to the frequency (eg. 8743064 ), I'd need a screen the size of an aircraft carrier.

    Even using a volume -- a sphere will have a radius of 127; a cube will be sized 200x200x200 -- that single point would fill the entire plot posted above.

  • Problem 2. Obscuration.

    No matter how you plot it -- even using (say) cubes of the logs of the frequencies; and expanding the XYZ scales by (say) 10 to create more space between the points -- no 3D to 2D plot allows you to see what is going on.

• Prettiness is not a consideration.

  I am the only one who will see these plots, so utility is far more important that prettiness.

• The plots are just a step on a road, not an end.

  I have a bunch of images, some quite large, and I'm looking to quickly visualise those datasets in order to explore possibilities for categorising them. Ie. The plots are just a way to look for how to tackle a problem, not a solution in themselves.

  They are throw away steps on the way to appreciating a bigger problem, and as such should not require large amounts of effort to develop.

In the past I've been lucky enough to have access to high-end proprietary software than would allow you to rotate a 3D plots on screen with the mouse or keyboard in real-time, but I don't know of any free tool that allows this. Nor is there any graphical toolkit (for Perl or anything else), that I'm aware of that would allow me to 'knock up' such an application quickly and easily.

The point here is that this visualisation is not the underlying problem I exploring. Just a step along the way to trying to get to grips with understanding a dataset. Whilst the particular sample image appears -- yet to be confirmed -- to have its pixels clustered both by frequency and HSV into a small number of distinct groups; it may be that when I apply this process to other relevant images, no such clustering occurs. So the goal is not to develop an all singing and dancing 4D data visualisation tool. It is to find a way to visualise a few example sets of data, in a few different ways, to see if there is anything there worth exploring.

Update: Here is another view. With the points plotted as circles using log2( freq ), and the image rotated so that V->X, S->Y, H->Z. It is interesting because it highlights the presence of more than two groups when viewed from this angle.

It is also disappointing because whilst it is easy enough to pick out the bigger sploges, could anyone pick out the 8.7 million splog versus the 1.5 million? Or even the 600,000?

Though this pure Tk.yellowstone.earthquake.gnuplot2.pl doesn't do rotation

Gnuplot allows you to rotate it, see Earthquakes in three dimensions

I'm not sure, but I think I also saw a Prima demo do 3d-ish rotation

Gnuplot is the tool I need. Just a shame it is so badly documented. (And that the Win32 port has limitations :( )

The rotating 3D earthquakes demo is ... um .. not well programmed. There is no need to reload the data for each rotation. That's what makes it so flickery. Here's my sample data rotating in X & Y using refresh:

#! perl -slw
use strict;
use warnings;
use IO::Handle;

my @data          = <DATA>;
my $fh   = IO::Handle->new;
$fh->autoflush(1);
open $fh, '|- ', 'c:\Downloaded\gp45-winbin\gnuplot\binary\\gnuplot'
    or die( "Could not open a pipe to gnuplot:$!");

print $fh "splot '-' using 1:2:3";
print $fh join('',@data);
print $fh "\ne";

for( my $xrot = 0, my $yrot = 0; 
    $xrot < 180; 
    $xrot += 0.5, $yrot = ( $yrot + 10 ) % 360 
) {
    print $fh "set view $xrot,$yrot";
    print $fh "refresh";
    Win32::Sleep( 100 );
}

print $fh "\nexit";
close($fh);

exit;

__DATA__
0.00108225108225108 0.639004149377593   0.945098039215686      8572
0.00265957446808511 0.899521531100478   0.819607843137255     11578
0.00349650349650348 0.671361502347418   0.835294117647059      5505
0.00735294117647056 0.272               0.980392156862745     23112
0.0101851851851852  0.861244019138756   0.819607843137255      9083
0.0120898100172712  0.824786324786325   0.917647058823529      4348
0.0126984126984127  0.421686746987952   0.976470588235294     25642
0.0145348837209302  0.950276243093923   0.709803921568627      6622
0.0145985401459854  0.643192488262911   0.835294117647059      5179
0.0166666666666667  0.540540540540541   0.435294117647059       345
0.0175438596491228  0.633333333333333   0.941176470588235      8163
0.0181818181818182  0.22                0.980392156862745     20719
0.0193236714975845  0.945205479452055   0.572549019607843      3602
0.0204081632653061  0.439461883408072   0.874509803921569      8518
0.020952380952381   0.837320574162679   0.819607843137255      7243
0.0252525252525252  0.916666666666667   0.705882352941177      3279
0.0267702936096718  0.824786324786325   0.917647058823529      2215
0.0296296296296296  0.405405405405405   0.435294117647059       472
0.0298507462686567  0.465277777777778   0.564705882352941       849
0.0303030303030303  0.622641509433962   0.831372549019608      4776
0.0348837209302326  0.826923076923077   0.815686274509804      4677
0.0350877192982456  0.633333333333333   0.941176470588235      8137
0.037712895377129   0.938356164383562   0.572549019607843       750
0.0377358490566038  0.888268156424581   0.701960784313725      2935
0.0381944444444444  0.2152466367713     0.874509803921569      5620
0.0384615384615385  0.419354838709677   0.972549019607843     21785
0.040587219343696   0.824786324786325   0.917647058823529      1777
0.0416666666666667  0.414414414414414   0.870588235294118      8993
0.0424836601307189  0.356643356643357   0.56078431372549        934
0.0460526315789474  0.617886178861789   0.964705882352941      9706
0.0490956072351421  0.611374407582938   0.827450980392157      7068
0.0501089324618736  0.854748603351955   0.701960784313725      2972
0.0503875968992248  0.826923076923077   0.815686274509804      3774
0.0508130081300813  0.460674157303371   0.698039215686274      2540
0.0516666666666667  0.909090909090909   0.431372549019608      3279
0.0517241379310345  0.371794871794872   0.305882352941176       795
0.0539215686274509  0.151785714285714   0.87843137254902       4558
0.0555555555555556  0.204819277108434   0.976470588235294     19783
0.0559701492537313  0.924137931034483   0.568627450980392       934
0.0563973063973064  0.842553191489362   0.92156862745098       1302
0.0606060606060606  0.6                 0.862745098039216      6506
0.0632183908045977  0.391891891891892   0.870588235294118      9557
0.0633608815426997  0.675977653631285   0.701960784313725      2657
0.0649350649350649  0.626016260162602   0.964705882352941     11189
0.0657596371882086  0.830508474576271   0.694117647058824      2901
0.065891472868217   0.826923076923077   0.815686274509804      3622
0.0666666666666667  0.425101214574899   0.968627450980392     31780
0.0731707317073171  0.366071428571429   0.43921568627451        473
0.0736434108527132  0.895833333333333   0.564705882352941      1587
0.0739837398373984  0.872340425531915   0.92156862745098       1625
0.0768115942028985  0.580808080808081   0.776470588235294      5220
0.0800653594771242  0.927272727272727   0.431372549019608      1297
0.0808080808080808  0.6                 0.862745098039216      8592
0.0823754789272031  0.836538461538462   0.815686274509804      3300
0.0849462365591398  0.63265306122449    0.96078431372549      14943
0.0854341736694678  0.668539325842697   0.698039215686274      2469
0.0876288659793814  0.392712550607287   0.968627450980392     48496
0.0882800608828006  0.9125              0.941176470588235      1663
0.0885416666666666  0.323232323232323   0.776470588235294      4653
0.0891472868217054  0.593103448275862   0.568627450980392       882
0.0917721518987342  0.902857142857143   0.686274509803922      2340
0.0919540229885057  0.390134529147982   0.874509803921569     13065
0.0933333333333333  0.301204819277108   0.325490196078431       320
0.0934959349593496  0.241176470588235   0.666666666666667      2510
0.0939153439153439  0.881118881118881   0.56078431372549       2391
0.0948717948717949  0.882352941176471   0.866666666666667      2991
0.0966666666666667  0.2                 0.980392156862745      8454
0.0967741935483871  0.264957264957265   0.458823529411765       535
0.101449275362319   0.583756345177665   0.772549019607843      6647
0.102852852852853   0.928870292887029   0.937254901960784      4163
0.103233830845771   0.609090909090909   0.862745098039216     13897
0.103260869565217   0.538011695906433   0.670588235294118      2820
0.104430379746835   0.647540983606557   0.956862745098039     36942
0.106060606060606   0.146017699115044   0.886274509803922      3453
0.106060606060606   0.908256880733945   0.427450980392157      2465
0.108877721943049   0.896396396396396   0.870588235294118      5918
0.109126984126984   0.861538461538462   0.764705882352941      4220
0.111111111111111   0.913793103448276   0.682352941176471      3887
0.112633181126332   0.916317991631799   0.937254901960784     10452
0.116504854368932   0.417004048582996   0.968627450980392    111192
0.116666666666667   0.357142857142857   0.87843137254902      20362
0.117370892018779   0.91025641025641    0.305882352941176      3654
0.117424242424242   0.602739726027397   0.572549019607843      1471
0.119047619047619   0.93006993006993    0.56078431372549       3740
0.122852233676976   0.881818181818182   0.862745098039216     11931
0.123501199040767   0.6289592760181     0.866666666666667     27107
0.124012638230648   0.879166666666667   0.941176470588235     30785
0.124203821656051   0.646090534979424   0.952941176470588    196031
0.124293785310734   0.595959595959596   0.776470588235294      8581
0.125968992248062   0.882051282051282   0.764705882352941      8786
0.129084967320261   0.918918918918919   0.435294117647059      3136
0.130952380952381   0.351758793969849   0.780392156862745      4884
0.132768361581921   0.51304347826087    0.450980392156863       391
0.133109619686801   0.892215568862275   0.654901960784314      7457
0.134488448844884   0.834710743801653   0.949019607843137    233164
0.136284722222222   0.872727272727273   0.862745098039216     39254
0.137345679012346   0.433734939759036   0.976470588235294     79008
0.1375              0.229885057471264   0.682352941176471      1401
0.13768115942029    0.544378698224852   0.662745098039216      4324
0.137844611528822   0.93006993006993    0.56078431372549       5298
0.139088729016787   0.620535714285714   0.87843137254902      43326
0.139227642276423   0.669387755102041   0.96078431372549     828431
0.13963963963964    0.157446808510638   0.92156862745098       3124
0.14                0.903614457831325   0.325490196078431      2397
0.140562248995984   0.367256637168142   0.886274509803922     31387
0.140655105973025   0.88265306122449    0.768627450980392     16813
0.144670050761421   0.807377049180328   0.956862745098039    902210
0.146953405017921   0.834080717488789   0.874509803921569     76370
0.148484848484849   0.668016194331984   0.968627450980392    691369
0.148611111111111   0.609137055837564   0.772549019607843      9144
0.149122807017544   0.791666666666667   0.0941176470588235   122575
0.149691358024691   0.939130434782609   0.450980392156863      3211
0.150793650793651   0.880239520958084   0.654901960784314     14022
0.151515151515152   0.801619433198381   0.968627450980392    471546
0.151724137931034   0.638766519823789   0.890196078431373    196584
0.153100775193798   0.868686868686869   0.776470588235294     19061
0.153846153846154   0.292134831460674   0.349019607843137       289
0.154054054054054   0.814977973568282   0.890196078431373    191506
0.154867256637168   0.448412698412698   0.988235294117647     13775
0.158486707566462   0.646825396825397   0.988235294117647     57912
0.158687943262411   0.796610169491525   0.925490196078431    316040
0.15993265993266    0.788844621513944   0.984313725490196     47824
0.16                0.892857142857143   0.219607843137255      5354
0.161417322834646   0.900709219858156   0.552941176470588      5322
0.16358024691358    0.81                0.784313725490196     14134
0.164556962025316   0.908045977011494   0.341176470588235      1271
0.16468253968254    0.60431654676259    0.545098039215686      2769
0.165242165242165   0.587939698492462   0.780392156862745      6869
0.165492957746479   0.840236686390533   0.662745098039216     14981
0.170175438596491   0.562130177514793   0.662745098039216      7405
0.173694779116466   0.786729857819905   0.827450980392157      8020
0.175736961451247   0.807692307692308   0.713725490196078      7516
0.177165354330709   0.601895734597156   0.827450980392157      6390
0.179012345679012   0.892561983471074   0.474509803921569      3319
0.180661577608142   0.861842105263158   0.596078431372549      8089
0.181957186544342   0.595628415300546   0.717647058823529      3723
0.185810810810811   0.758974358974359   0.764705882352941      2655
0.188679245283019   0.291208791208791   0.713725490196078      1162
0.189003436426117   0.633986928104575   0.6                    6121
0.191964285714286   0.574358974358974   0.764705882352941      1896
0.192857142857143   0.573770491803279   0.47843137254902       1275
0.197222222222222   0.72289156626506    0.650980392156863      6799
0.198005698005698   0.860294117647059   0.533333333333333      4789
0.207671957671958   0.84                0.588235294117647      3288
0.209677419354839   0.885714285714286   0.411764705882353       943
0.211382113821138   0.687150837988827   0.701960784313725      2100
0.215447154471545   0.598540145985401   0.537254901960784      2912
0.225177304964539   0.622516556291391   0.592156862745098      2400
0.225490196078431   0.85                0.470588235294118      1576
0.227272727272727   0.674846625766871   0.63921568627451       1917
0.236286919831224   0.441340782122905   0.701960784313725      1187
0.246753246753247   0.865168539325843   0.349019607843137       302
0.247368421052632   0.708955223880597   0.525490196078431      1736
0.248484848484849   0.514018691588785   0.419607843137255       306
0.259803921568627   0.56198347107438    0.474509803921569       690
0.267326732673267   0.668874172185431   0.592156862745098       638
0.270325203252033   0.788461538461538   0.407843137254902       292
0.289156626506024   0.703389830508475   0.462745098039216       457
0.315789473684211   0.853932584269663   0.349019607843137       164
0.323899371069182   0.389705882352941   0.533333333333333       618
0.333333333333333   0.130434782608696   0.270588235294118       158
0.403846153846154   0.490566037735849   0.415686274509804       110
0.471014492753623   0.165467625899281   0.545098039215686       578
0.471264367816092   0.386666666666667   0.588235294117647        42
0.888888888888889   0.916666666666667   0.988235294117647    162911
0.908232118758435   0.98015873015873    0.988235294117647   8743064
0.916302765647744   0.908730158730159   0.988235294117647    158599
0.919444444444444   0.714285714285714   0.988235294117647      8648
0.924100156494523   0.845238095238095   0.988235294117647     68132
0.926829268292683   0.650793650793651   0.988235294117647      8169
0.931472081218274   0.781746031746032   0.988235294117647     35059
0.932489451476793   0.741784037558685   0.835294117647059      1442
0.936936936936937   0.587301587301587   0.988235294117647      9611
0.943105756358768   0.988095238095238   0.988235294117647   1554975
0.944927536231884   0.456349206349206   0.988235294117647      9620
0.94537037037037    0.714285714285714   0.988235294117647     22456
0.947619047619048   0.972222222222222   0.705882352941177     23184
0.948232323232323   0.523809523809524   0.988235294117647     11920
0.95007342143906    0.908               0.980392156862745     41821
0.955082742316785   0.661971830985916   0.835294117647059      1845
0.956032719836401   0.649402390438247   0.984313725490196     19910
0.960737179487179   0.838709677419355   0.972549019607843     18883
0.970319634703196   0.584               0.980392156862745     17791
0.971052631578947   0.772357723577236   0.964705882352941     14649
0.973544973544973   0.580645161290323   0.850980392156863      3339
0.974074074074074   0.76271186440678    0.694117647058824      1020
0.981707317073171   0.326693227091633   0.984313725490196     18462
0.984159779614325   0.991803278688525   0.956862745098039    170398
0.98635477582846    0.703703703703704   0.952941176470588      9256
0.986979166666667   0.516129032258065   0.972549019607843     17392
0.989827856025039   0.902542372881356   0.925490196078431     11411
0.990909090909091   0.509259259259259   0.847058823529412      4854
0.991228070175439   0.456               0.980392156862745     18555
0.993589743589744   0.233183856502242   0.874509803921569      4207
0.997572815533981   0.985645933014354   0.819607843137255    110786
0                   0.597222222222222   0.564705882352941       430
0                   0.833333333333333   0.917647058823529      6549
0                   0                   1                      8453
[download]

Just shame I can't get it to use variable sized circles instead of crosses?

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

I've made simple 3d plots of your data mapping H,V,F and H,S,F and they show interesting clustering.

I think reducing the number of points will help, I created a grid 100 x 100 and summed all the frequencies of points within each cell.

I've never seen anything like that, but the only thing I can come up with is to draw a series of 2D frequency contour maps each for a given hue and stack them in some way.

or scroll through them like rolodex cards, if you want to get fancy ;)

Cool question! I'd use voxels, or more specifically---doxels, to get the job done. Graphics::VTK has some support for voxels. It requires The Visualization ToolKit. And for some quick examples, there's the Voxlap Engine and download here.

See also:

I'm not sure i completly follow. But i saw something like that realized as some kind of LED based audio visualizer: YouTube Video

how to represent the frequency at that point

A more serious note, one way to visualize 6D data in 2D is to use coloured disks: 3 dimensions for its colour, 2 dimensions for the position of its center, and using the radius gives you a sixth.

Nice. The circular sector can give you the seventh :)