Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw

Quick script to check data logger data

by GrandFather (Saint)
on Feb 10, 2021 at 09:54 UTC ( #11128177=CUFP: print w/replies, xml ) Need Help??

The coolness factor in this script is more to do with the utility of Perl for whipping up a tool than very much that is special in the script. That said, the script does demonstrate why Perl is a great "whip it up language".

The back story is that the astronomical society I'm a member of hosts a number of space weather related data collection systems at our dark sky site. The systems belong to researchers in Japan and the instruments we operate are part of an international network of similar instruments. About a year ago we added an induction magnetometer to the collection of instruments. Unfortunately it has been nothing but trouble. Over the period of a year we got about three days of data off it. Long story short - there was a hardware fault with the data logger. I've fixed that and now we are starting to collect data so I needed a way to check that the data is sensible. Enter Perl.

As it happens the file format for the data is documented - as a C++ header containing a couple of C++ structs. That's OK. Perl is quite happy to read fixed size chunks from files then convert the chunks using unpack. Turned out that for the current task I needn't have bothered, but I'll use that code for another task in the future anyway. The data itself is kinda meaningless. It becomes interesting when passed through a FFT (Fast Fourier Transform). Hey, what do ya know, CPAN has a FFT module (several actually). And the data from an FFT is kinda unintelligible without plotting it - Tk, please step up.

So the following (abridged) script is the result of a couple of hours work and lets me drag and drop files off the data logger onto a shortcut and make a visual check that the data looks sane.

use strict; use warnings; use Math::FFT; use Tk; use Tk::Canvas; package ATSComments80_s; sub new { my ($class, $fileHandle) = @_; my $self = bless {fh => $fileHandle}, $class; $self->load(); return $self; } sub load { my ($self) = @_; read $self->{fh}, $self->{achClient}, 16; read $self->{fh}, $self->{achContractor}, 16; read $self->{fh}, $self->{achArea}, 16; read $self->{fh}, $self->{achSurveyID}, 16; read $self->{fh}, $self->{achOperator}, 16; read $self->{fh}, $self->{achReserved}, 112; read $self->{fh}, $self->{achXmlHeader}, 64; read $self->{fh}, $self->{achComments}, 512; } package ATSHeader80_s; sub new { my ($class, $fileHandle) = @_; my $self = bless {fh => $fileHandle}, $class; $self->load(); return $self; } sub load { my ($self) = @_; read $self->{fh}, $self->{siHeaderLength}, 2; $self->{siHeaderLength} = unpack 's<', $self->{siHeaderLength}; ...; read $self->{fh}, $self->{abyADBBoardType}, 4; $self->{tscComment} = ATSComments80_s->new($self->{fh}); } package main; my $filePath = $ARGV[0] // "testData.ats"; open my $fh, '<:raw', $filePath or die "Can't open $filePath: $!"; my $header = ATSHeader80_s->new($fh); my $itemCount = 0; my @data; while ($itemCount++ < 1024 && read $fh, my ($value), 4) { $value = unpack 'l<', $value; push @data, $value; } my $fft = Math::FFT->new(\@data); my $mw = MainWindow->new (-title => "Magnetometer Plotter"); my $canvas = $mw->Canvas (-height => 700, -width => 1024)->pack (); my $spectrum = $fft->spctrm; # Remove DC signal component shift @$spectrum; $spectrum = NormData($spectrum, 680); $canvas->createLine( (map {2 + 2 * $_, $spectrum->[$_]} 0 .. $#$spectrum), -fill => 'blue' ); $mw->MainLoop (); sub NormData { my ($data, $span) = @_; my $min; my $max; for my $datum (@$data) { $min //= $datum; $max //= $datum; $min = $datum if $min > $datum; $max = $datum if $max < $datum; } my $scale = $span / ($max - $min); $_ = $span - ($_ - $min) * $scale + 10 for @$data; return $data; }
Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond

Replies are listed 'Best First'.
Re: Quick script to check data logger data
by jdporter (Canon) on Feb 10, 2021 at 17:41 UTC


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: CUFP [id://11128177]
Approved by marto
Front-paged by marto
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (2)
As of 2021-04-13 03:23 GMT
Find Nodes?
    Voting Booth?

    No recent polls found