Re: [OT] Displaying 4D data in a 2D image. (Reply to all respondants. (Thank you!) :)

Okay. I guess I should have supplied some data and some explanation of what I've tried to date. (I didn't want to influence responses or cloud the underlying question with specifics of how and with what tools to do the actual drawing.)

Here is an illustrative, though rather small set of sample data. The keys are HSV (all scaled 0 .. 1 rather than H being 0 .. 360 ). The values are the frequencies of pixels with that HSV value within the image being analysed:

And here is a 2D/3D representation of that dataset. The H,S,V -> X,Y,Z in a 256x256x256 'cube'. The color of the plotted points is the pixel color.

As you can see, the HSV space discriminates the points into two very clear clusters when viewed this way. However, if viewed from different angles, it is possible to discern 5 or even eight clusterings with a few outliers.

If you look at the frequency data, it is possible to also find two very obvious peaks; and 4 or 5 or 7 more, depending where you apply the cutoff. The challenge is to try and a) find that cut-off; b) partition the dataset around those peaks.

I've made no attempt in this image to plot the frequency. The problem with trying to represent the frequency is the range of the values -- from 1 up to 8.7 million. It could be much bigger for larger images.

Using color: No color ramp can cover this kind of range.
Even if I used logarithms of the actual frequencies, the human brain is not well adapted to intuiting peaks and troughs of this magnitude from single points of color, let alone if they are logarithmic. Color ramps work well for surfaces where you can visualise darker colors as 'low' and brighter colors as 'high', because of the flow (gradation) between them.
Not so good for this though.
Using the size of the pixel/circle/voxel drawn within the 3D space.
- Problem 1. Obliteration.
  If I draw the 'point/circle/volume' directly proportional to the frequency (eg. 8743064 ), I'd need a screen the size of an aircraft carrier.
  Even using a volume -- a sphere will have a radius of 127; a cube will be sized 200x200x200 -- that single point would fill the entire plot posted above.
- Problem 2. Obscuration.
  No matter how you plot it -- even using (say) cubes of the logs of the frequencies; and expanding the XYZ scales by (say) 10 to create more space between the points -- no 3D to 2D plot allows you to see what is going on.
Prettiness is not a consideration.
I am the only one who will see these plots, so utility is far more important that prettiness.
The plots are just a step on a road, not an end.
I have a bunch of images, some quite large, and I'm looking to quickly visualise those datasets in order to explore possibilities for categorising them. Ie. The plots are just a way to look for how to tackle a problem, not a solution in themselves.
They are throw away steps on the way to appreciating a bigger problem, and as such should not require large amounts of effort to develop.

In the past I've been lucky enough to have access to high-end proprietary software than would allow you to rotate a 3D plots on screen with the mouse or keyboard in real-time, but I don't know of any free tool that allows this. Nor is there any graphical toolkit (for Perl or anything else), that I'm aware of that would allow me to 'knock up' such an application quickly and easily.

The point here is that this visualisation is not the underlying problem I exploring. Just a step along the way to trying to get to grips with understanding a dataset. Whilst the particular sample image appears -- yet to be confirmed -- to have its pixels clustered both by frequency and HSV into a small number of distinct groups; it may be that when I apply this process to other relevant images, no such clustering occurs. So the goal is not to develop an all singing and dancing 4D data visualisation tool. It is to find a way to visualise a few example sets of data, in a few different ways, to see if there is anything there worth exploring.

Update: Here is another view. With the points plotted as circles using log2( freq ), and the image rotated so that V->X, S->Y, H->Z. It is interesting because it highlights the presence of more than two groups when viewed from this angle.

It is also disappointing because whilst it is easy enough to pick out the bigger sploges, could anyone pick out the 8.7 million splog versus the 1.5 million? Or even the 600,000?

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

Comment on Re: [OT] Displaying 4D data in a 2D image. (Reply to all respondants. (Thank you!) :) Download Code

Replies are listed 'Best First'.
Re^2: [OT] Displaying 4D data in a 2D image. (Reply to all respondants. (Thank you!) :) by Anonymous Monk on Dec 10, 2011 at 04:53 UTC
Though this pure Tk.yellowstone.earthquake.gnuplot2.pl doesn't do rotation Gnuplot allows you to rotate it, see Earthquakes in three dimensions I'm not sure, but I think I also saw a Prima demo do 3d-ish rotation	[reply]
Re^3: [OT] Displaying 4D data in a 2D image. (Reply to all respondants. (Thank you!) :) by BrowserUk (Patriarch) on Dec 10, 2011 at 09:44 UTC
Gnuplot is the tool I need. Just a shame it is so badly documented. (And that the Win32 port has limitations :( ) The rotating 3D earthquakes demo is ... um .. not well programmed. There is no need to reload the data for each rotation. That's what makes it so flickery. Here's my sample data rotating in X & Y using refresh: #! perl -slw use strict; use warnings; use IO::Handle; my @data = <DATA>; my $fh = IO::Handle->new; $fh->autoflush(1); open $fh, '\|- ', 'c:\Downloaded\gp45-winbin\gnuplot\binary\\gnuplot' or die( "Could not open a pipe to gnuplot:$!"); print $fh "splot '-' using 1:2:3"; print $fh join('',@data); print $fh "\ne"; for( my $xrot = 0, my $yrot = 0; $xrot < 180; $xrot += 0.5, $yrot = ( $yrot + 10 ) % 360 ) { print $fh "set view $xrot,$yrot"; print $fh "refresh"; Win32::Sleep( 100 ); } print $fh "\nexit"; close($fh); exit; __DATA__ 0.00108225108225108 0.639004149377593 0.945098039215686 8572 0.00265957446808511 0.899521531100478 0.819607843137255 11578 0.00349650349650348 0.671361502347418 0.835294117647059 5505 0.00735294117647056 0.272 0.980392156862745 23112 0.0101851851851852 0.861244019138756 0.819607843137255 9083 0.0120898100172712 0.824786324786325 0.917647058823529 4348 0.0126984126984127 0.421686746987952 0.976470588235294 25642 0.0145348837209302 0.950276243093923 0.709803921568627 6622 0.0145985401459854 0.643192488262911 0.835294117647059 5179 0.0166666666666667 0.540540540540541 0.435294117647059 345 0.0175438596491228 0.633333333333333 0.941176470588235 8163 0.0181818181818182 0.22 0.980392156862745 20719 0.0193236714975845 0.945205479452055 0.572549019607843 3602 0.0204081632653061 0.439461883408072 0.874509803921569 8518 0.020952380952381 0.837320574162679 0.819607843137255 7243 0.0252525252525252 0.916666666666667 0.705882352941177 3279 0.0267702936096718 0.824786324786325 0.917647058823529 2215 0.0296296296296296 0.405405405405405 0.435294117647059 472 0.0298507462686567 0.465277777777778 0.564705882352941 849 0.0303030303030303 0.622641509433962 0.831372549019608 4776 0.0348837209302326 0.826923076923077 0.815686274509804 4677 0.0350877192982456 0.633333333333333 0.941176470588235 8137 0.037712895377129 0.938356164383562 0.572549019607843 750 0.0377358490566038 0.888268156424581 0.701960784313725 2935 0.0381944444444444 0.2152466367713 0.874509803921569 5620 0.0384615384615385 0.419354838709677 0.972549019607843 21785 0.040587219343696 0.824786324786325 0.917647058823529 1777 0.0416666666666667 0.414414414414414 0.870588235294118 8993 0.0424836601307189 0.356643356643357 0.56078431372549 934 0.0460526315789474 0.617886178861789 0.964705882352941 9706 0.0490956072351421 0.611374407582938 0.827450980392157 7068 0.0501089324618736 0.854748603351955 0.701960784313725 2972 0.0503875968992248 0.826923076923077 0.815686274509804 3774 0.0508130081300813 0.460674157303371 0.698039215686274 2540 0.0516666666666667 0.909090909090909 0.431372549019608 3279 0.0517241379310345 0.371794871794872 0.305882352941176 795 0.0539215686274509 0.151785714285714 0.87843137254902 4558 0.0555555555555556 0.204819277108434 0.976470588235294 19783 0.0559701492537313 0.924137931034483 0.568627450980392 934 0.0563973063973064 0.842553191489362 0.92156862745098 1302 0.0606060606060606 0.6 0.862745098039216 6506 0.0632183908045977 0.391891891891892 0.870588235294118 9557 0.0633608815426997 0.675977653631285 0.701960784313725 2657 0.0649350649350649 0.626016260162602 0.964705882352941 11189 0.0657596371882086 0.830508474576271 0.694117647058824 2901 0.065891472868217 0.826923076923077 0.815686274509804 3622 0.0666666666666667 0.425101214574899 0.968627450980392 31780 0.0731707317073171 0.366071428571429 0.43921568627451 473 0.0736434108527132 0.895833333333333 0.564705882352941 1587 0.0739837398373984 0.872340425531915 0.92156862745098 1625 0.0768115942028985 0.580808080808081 0.776470588235294 5220 0.0800653594771242 0.927272727272727 0.431372549019608 1297 0.0808080808080808 0.6 0.862745098039216 8592 0.0823754789272031 0.836538461538462 0.815686274509804 3300 0.0849462365591398 0.63265306122449 0.96078431372549 14943 0.0854341736694678 0.668539325842697 0.698039215686274 2469 0.0876288659793814 0.392712550607287 0.968627450980392 48496 0.0882800608828006 0.9125 0.941176470588235 1663 0.0885416666666666 0.323232323232323 0.776470588235294 4653 0.0891472868217054 0.593103448275862 0.568627450980392 882 0.0917721518987342 0.902857142857143 0.686274509803922 2340 0.0919540229885057 0.390134529147982 0.874509803921569 13065 0.0933333333333333 0.301204819277108 0.325490196078431 320 0.0934959349593496 0.241176470588235 0.666666666666667 2510 0.0939153439153439 0.881118881118881 0.56078431372549 2391 0.0948717948717949 0.882352941176471 0.866666666666667 2991 0.0966666666666667 0.2 0.980392156862745 8454 0.0967741935483871 0.264957264957265 0.458823529411765 535 0.101449275362319 0.583756345177665 0.772549019607843 6647 0.102852852852853 0.928870292887029 0.937254901960784 4163 0.103233830845771 0.609090909090909 0.862745098039216 13897 0.103260869565217 0.538011695906433 0.670588235294118 2820 0.104430379746835 0.647540983606557 0.956862745098039 36942 0.106060606060606 0.146017699115044 0.886274509803922 3453 0.106060606060606 0.908256880733945 0.427450980392157 2465 0.108877721943049 0.896396396396396 0.870588235294118 5918 0.109126984126984 0.861538461538462 0.764705882352941 4220 0.111111111111111 0.913793103448276 0.682352941176471 3887 0.112633181126332 0.916317991631799 0.937254901960784 10452 0.116504854368932 0.417004048582996 0.968627450980392 111192 0.116666666666667 0.357142857142857 0.87843137254902 20362 0.117370892018779 0.91025641025641 0.305882352941176 3654 0.117424242424242 0.602739726027397 0.572549019607843 1471 0.119047619047619 0.93006993006993 0.56078431372549 3740 0.122852233676976 0.881818181818182 0.862745098039216 11931 0.123501199040767 0.6289592760181 0.866666666666667 27107 0.124012638230648 0.879166666666667 0.941176470588235 30785 0.124203821656051 0.646090534979424 0.952941176470588 196031 0.124293785310734 0.595959595959596 0.776470588235294 8581 0.125968992248062 0.882051282051282 0.764705882352941 8786 0.129084967320261 0.918918918918919 0.435294117647059 3136 0.130952380952381 0.351758793969849 0.780392156862745 4884 0.132768361581921 0.51304347826087 0.450980392156863 391 0.133109619686801 0.892215568862275 0.654901960784314 7457 0.134488448844884 0.834710743801653 0.949019607843137 233164 0.136284722222222 0.872727272727273 0.862745098039216 39254 0.137345679012346 0.433734939759036 0.976470588235294 79008 0.1375 0.229885057471264 0.682352941176471 1401 0.13768115942029 0.544378698224852 0.662745098039216 4324 0.137844611528822 0.93006993006993 0.56078431372549 5298 0.139088729016787 0.620535714285714 0.87843137254902 43326 0.139227642276423 0.669387755102041 0.96078431372549 828431 0.13963963963964 0.157446808510638 0.92156862745098 3124 0.14 0.903614457831325 0.325490196078431 2397 0.140562248995984 0.367256637168142 0.886274509803922 31387 0.140655105973025 0.88265306122449 0.768627450980392 16813 0.144670050761421 0.807377049180328 0.956862745098039 902210 0.146953405017921 0.834080717488789 0.874509803921569 76370 0.148484848484849 0.668016194331984 0.968627450980392 691369 0.148611111111111 0.609137055837564 0.772549019607843 9144 0.149122807017544 0.791666666666667 0.0941176470588235 122575 0.149691358024691 0.939130434782609 0.450980392156863 3211 0.150793650793651 0.880239520958084 0.654901960784314 14022 0.151515151515152 0.801619433198381 0.968627450980392 471546 0.151724137931034 0.638766519823789 0.890196078431373 196584 0.153100775193798 0.868686868686869 0.776470588235294 19061 0.153846153846154 0.292134831460674 0.349019607843137 289 0.154054054054054 0.814977973568282 0.890196078431373 191506 0.154867256637168 0.448412698412698 0.988235294117647 13775 0.158486707566462 0.646825396825397 0.988235294117647 57912 0.158687943262411 0.796610169491525 0.925490196078431 316040 0.15993265993266 0.788844621513944 0.984313725490196 47824 0.16 0.892857142857143 0.219607843137255 5354 0.161417322834646 0.900709219858156 0.552941176470588 5322 0.16358024691358 0.81 0.784313725490196 14134 0.164556962025316 0.908045977011494 0.341176470588235 1271 0.16468253968254 0.60431654676259 0.545098039215686 2769 0.165242165242165 0.587939698492462 0.780392156862745 6869 0.165492957746479 0.840236686390533 0.662745098039216 14981 0.170175438596491 0.562130177514793 0.662745098039216 7405 0.173694779116466 0.786729857819905 0.827450980392157 8020 0.175736961451247 0.807692307692308 0.713725490196078 7516 0.177165354330709 0.601895734597156 0.827450980392157 6390 0.179012345679012 0.892561983471074 0.474509803921569 3319 0.180661577608142 0.861842105263158 0.596078431372549 8089 0.181957186544342 0.595628415300546 0.717647058823529 3723 0.185810810810811 0.758974358974359 0.764705882352941 2655 0.188679245283019 0.291208791208791 0.713725490196078 1162 0.189003436426117 0.633986928104575 0.6 6121 0.191964285714286 0.574358974358974 0.764705882352941 1896 0.192857142857143 0.573770491803279 0.47843137254902 1275 0.197222222222222 0.72289156626506 0.650980392156863 6799 0.198005698005698 0.860294117647059 0.533333333333333 4789 0.207671957671958 0.84 0.588235294117647 3288 0.209677419354839 0.885714285714286 0.411764705882353 943 0.211382113821138 0.687150837988827 0.701960784313725 2100 0.215447154471545 0.598540145985401 0.537254901960784 2912 0.225177304964539 0.622516556291391 0.592156862745098 2400 0.225490196078431 0.85 0.470588235294118 1576 0.227272727272727 0.674846625766871 0.63921568627451 1917 0.236286919831224 0.441340782122905 0.701960784313725 1187 0.246753246753247 0.865168539325843 0.349019607843137 302 0.247368421052632 0.708955223880597 0.525490196078431 1736 0.248484848484849 0.514018691588785 0.419607843137255 306 0.259803921568627 0.56198347107438 0.474509803921569 690 0.267326732673267 0.668874172185431 0.592156862745098 638 0.270325203252033 0.788461538461538 0.407843137254902 292 0.289156626506024 0.703389830508475 0.462745098039216 457 0.315789473684211 0.853932584269663 0.349019607843137 164 0.323899371069182 0.389705882352941 0.533333333333333 618 0.333333333333333 0.130434782608696 0.270588235294118 158 0.403846153846154 0.490566037735849 0.415686274509804 110 0.471014492753623 0.165467625899281 0.545098039215686 578 0.471264367816092 0.386666666666667 0.588235294117647 42 0.888888888888889 0.916666666666667 0.988235294117647 162911 0.908232118758435 0.98015873015873 0.988235294117647 8743064 0.916302765647744 0.908730158730159 0.988235294117647 158599 0.919444444444444 0.714285714285714 0.988235294117647 8648 0.924100156494523 0.845238095238095 0.988235294117647 68132 0.926829268292683 0.650793650793651 0.988235294117647 8169 0.931472081218274 0.781746031746032 0.988235294117647 35059 0.932489451476793 0.741784037558685 0.835294117647059 1442 0.936936936936937 0.587301587301587 0.988235294117647 9611 0.943105756358768 0.988095238095238 0.988235294117647 1554975 0.944927536231884 0.456349206349206 0.988235294117647 9620 0.94537037037037 0.714285714285714 0.988235294117647 22456 0.947619047619048 0.972222222222222 0.705882352941177 23184 0.948232323232323 0.523809523809524 0.988235294117647 11920 0.95007342143906 0.908 0.980392156862745 41821 0.955082742316785 0.661971830985916 0.835294117647059 1845 0.956032719836401 0.649402390438247 0.984313725490196 19910 0.960737179487179 0.838709677419355 0.972549019607843 18883 0.970319634703196 0.584 0.980392156862745 17791 0.971052631578947 0.772357723577236 0.964705882352941 14649 0.973544973544973 0.580645161290323 0.850980392156863 3339 0.974074074074074 0.76271186440678 0.694117647058824 1020 0.981707317073171 0.326693227091633 0.984313725490196 18462 0.984159779614325 0.991803278688525 0.956862745098039 170398 0.98635477582846 0.703703703703704 0.952941176470588 9256 0.986979166666667 0.516129032258065 0.972549019607843 17392 0.989827856025039 0.902542372881356 0.925490196078431 11411 0.990909090909091 0.509259259259259 0.847058823529412 4854 0.991228070175439 0.456 0.980392156862745 18555 0.993589743589744 0.233183856502242 0.874509803921569 4207 0.997572815533981 0.985645933014354 0.819607843137255 110786 0 0.597222222222222 0.564705882352941 430 0 0.833333333333333 0.917647058823529 6549 0 0 1 8453 [download] Just shame I can't get it to use variable sized circles instead of crosses? With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply] [d/l]
Re^4: [OT] Displaying 4D data in a 2D image. (Reply to all respondants. (Thank you!) :) by Anonymous Monk on Dec 10, 2011 at 10:14 UTC
Just shame I can't get it to use variable sized circles instead of crosses? I tried `print $fh "set style data circles\n";` but my verion (4.3) doesn't support that, and I don't know if newer ones do. ?? http://www.gnuplot.info/demo/circles.html ?? http://www.gnuplotting.org/understand-parametric-plotting/	[reply] [d/l]
Re^5: [OT] Displaying 4D data in a 2D image. (Reply to all respondants. (Thank you!) :) by BrowserUk (Patriarch) on Dec 10, 2011 at 10:21 UTC
Re^2: [OT] Displaying 4D data in a 2D image. (Reply to all respondants. (Thank you!) :) by RichardK (Parson) on Dec 10, 2011 at 14:43 UTC
I've made simple 3d plots of your data mapping H,V,F and H,S,F and they show interesting clustering. I think reducing the number of points will help, I created a grid 100 x 100 and summed all the frequencies of points within each cell.	[reply]