The coolness factor in this script is more to do with the utility of Perl for whipping up a tool than very much that is special in the script. That said, the script does demonstrate why Perl is a great "whip it up language".

The back story is that the astronomical society I'm a member of hosts a number of space weather related data collection systems at our dark sky site. The systems belong to researchers in Japan and the instruments we operate are part of an international network of similar instruments. About a year ago we added an induction magnetometer to the collection of instruments. Unfortunately it has been nothing but trouble. Over the period of a year we got about three days of data off it. Long story short - there was a hardware fault with the data logger. I've fixed that and now we are starting to collect data so I needed a way to check that the data is sensible. Enter Perl.

As it happens the file format for the data is documented - as a C++ header containing a couple of C++ structs. That's OK. Perl is quite happy to read fixed size chunks from files then convert the chunks using unpack. Turned out that for the current task I needn't have bothered, but I'll use that code for another task in the future anyway. The data itself is kinda meaningless. It becomes interesting when passed through a FFT (Fast Fourier Transform). Hey, what do ya know, CPAN has a FFT module (several actually). And the data from an FFT is kinda unintelligible without plotting it - Tk, please step up.

So the following (abridged) script is the result of a couple of hours work and lets me drag and drop files off the data logger onto a shortcut and make a visual check that the data looks sane.

use strict; use warnings; use Math::FFT; use Tk; use Tk::Canvas; package ATSComments80_s; sub new { my ($class, $fileHandle) = @_; my $self = bless {fh => $fileHandle}, $class; $self->load(); return $self; } sub load { my ($self) = @_; read $self->{fh}, $self->{achClient}, 16; read $self->{fh}, $self->{achContractor}, 16; read $self->{fh}, $self->{achArea}, 16; read $self->{fh}, $self->{achSurveyID}, 16; read $self->{fh}, $self->{achOperator}, 16; read $self->{fh}, $self->{achReserved}, 112; read $self->{fh}, $self->{achXmlHeader}, 64; read $self->{fh}, $self->{achComments}, 512; } package ATSHeader80_s; sub new { my ($class, $fileHandle) = @_; my $self = bless {fh => $fileHandle}, $class; $self->load(); return $self; } sub load { my ($self) = @_; read $self->{fh}, $self->{siHeaderLength}, 2; $self->{siHeaderLength} = unpack 's<', $self->{siHeaderLength}; ...; read $self->{fh}, $self->{abyADBBoardType}, 4; $self->{tscComment} = ATSComments80_s->new($self->{fh}); } package main; my $filePath = $ARGV[0] // "testData.ats"; open my $fh, '<:raw', $filePath or die "Can't open $filePath: $!"; my $header = ATSHeader80_s->new($fh); my $itemCount = 0; my @data; while ($itemCount++ < 1024 && read $fh, my ($value), 4) { $value = unpack 'l<', $value; push @data, $value; } my $fft = Math::FFT->new(\@data); my $mw = MainWindow->new (-title => "Magnetometer Plotter"); my $canvas = $mw->Canvas (-height => 700, -width => 1024)->pack (); my $spectrum = $fft->spctrm; # Remove DC signal component shift @$spectrum; $spectrum = NormData($spectrum, 680); $canvas->createLine( (map {2 + 2 * $_, $spectrum->[$_]} 0 .. $#$spectrum), -fill => 'blue' ); $mw->MainLoop (); sub NormData { my ($data, $span) = @_; my $min; my $max; for my $datum (@$data) { $min //= $datum; $max //= $datum; $min = $datum if $min > $datum; $max = $datum if $max < $datum; } my $scale = $span / ($max - $min); $_ = $span - ($_ - $min) * $scale + 10 for @$data; return $data; }
Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond

Replies are listed 'Best First'.
Re: Quick script to check data logger data
by jdporter (Paladin) on Feb 10, 2021 at 17:41 UTC

    whipuptitude!

Re: Quick script to check data logger data
by etj (Priest) on Aug 08, 2024 at 15:23 UTC
    Astronomical... data... FFT... processing.... musn't... mention... PDL...

      I pick PDL up for tasks occasionally, but not often enough to maintain fluency with it. It's one of those catch-22 situations where PDL would be a really handy tool to have in the toolbox, but <fluency> * <need to use> never quite peeks over the <is more productive with> threshold. Just a little more <need to use> would increase <fluency> enough to make the difference, but I find too many other things to occupy my time so don't go out of my way enough to find excuses to use it.

      Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond
        I'd be interested to see a bit more concrete data and the problem you're solving, since I expect it would be easy for a journeyman like me to turn that into a couple of lines of PDL; that would then be a useful example I could then steal into some docs. Here, that would include a screenshot of how the Tk window actually looks, since PDL has some really good plotting stuff available.

      etj, i had a similar thought regarding Net::Clacks. I mean, it's a sensor that delivers data in real time. Not having that update up on some web page, also in real time (complete with timestamp of last sucessful read) would give me sleepless nights.

      Ok, yes, admittedly, it would also give me sleepless nights WITH real time monitoring. At least everytime that sensor runs into trouble, a red alarm light would light up on my big DIY LED status panel (next to my computer) and the master caution alarm would start to blare. "This thing watches for geomagnetic storms. If it fails, better assume that a large one is in progress and has taken out the sensor"(*).


      (*) "Fail save" wakes you up at night. "Fail deadly" keeps you asleep forever.

      PerlMonks XP is useless? Not anymore: XPD - Do more with your PerlMonks XP
      Also check out my sisters artwork and my weekly webcomics