Re^4: Quick script to check data logger data

Turns out when I said "now we are starting to collect data", I spoke too soon. There are bugs in the software that drive the logger and, after months of back and forth between us and the manufacturer, we haven't been able to get the system to work correctly.

The data consists of measurements in two axis from an induction magnetometer at 64Hz. The analysis looks for signals at fairly low frequency (7Hz maybe) that are related to some ionospheric phenomena I've forgotten about. Data files are a day's worth of data can come out to around 21MB. My test script looks at the first 16 seconds (1024 points) and plots a simple frequency/amplitude graph just to check that the data looks sane (basically that it has wiggly bits). The data below is fairly typical and shows a spike just below 8Hz - the interesting bit to the scientists. Absolute amplitude isn't very interesting for a quick assessment of the system.

-1054355 -934448 -1083864 -1289846 -1362502 -1251771 -956710 -826765 -
+1036961
-1107361 -861209 -822089 -1070997 -1151255 -1022497 -1007843 -1045329 
+-965611
-1006518 -1105524 -969199 -834501 -836961 -889544 -1083336 -1161387 -1
+013110
-1032829 -1164512 -1140949 -1048894 -950036 -982239 -1167254 -1152554 
+-1052439
-1157364 -1219241 -1084558 -914330 -959233 -1212299 -1235228 -1022051 
+-1043316
-1235664 -1215237 -1076013 -1097741 -1196784 -1130571 -950403 -986063 
+-1163735
-1073039 -987492 -1225996 -1301753 -1054177 -984974 -1118071 -1046077 
+-934674
-1002840 -1032935 -1170383 -1080441 -871046 -2101892 -3012085 -1410560
+ -144178
-426156 -584073 -714243 -1039425 -1129146 -1032533 -1098582 -1225863 -
+1092377
-1158012 -1348995 -1113281 -957586 -1204903 -1693565 -1845858 -1093056
+ -625118
-1010615 -987534 -867248 -1541651 -1733959 -846189 -425518 -1062193 -1
+599191
-1093092 -512186 -1157578 -1724370 -835494 -101072 -648677 -1262616 -1
+219483
-864121 -837411 -1277211 -1363956 -926084 -789910 -1048298 -1262266 -1
+364887
-988971 -1229990 -2729175 -1789255 1095947 896226 -1005292 -1448384 -1
+241273
-1157566 -973054 -1087618 -1647032 -1592968 -1049262 -1077479 -1141678
+ -1168128
-1489736 -298666 757862 -1191923 -2392586 -1298409 -740727 -938564 -11
+58873
-1243077 -1225406 -926746 -784887 -1305888 -1536915 -1224183 -1068087 
+-1132526
-1189950 -1129911 -1072430 -1214240 -1399874 -1359004 -1151840 -986897
+ -1026991
-1112164 -1087705 -1239457 -1426727 -1292056 -1001813 -806323 -824692 
+-996581
-1074320 -990701 -1078806 -1294623 -1150505 -996057 -1112120 -1026471 
+-898243
-1125728 -1402701 -1256597 -1021666 -743675 -484198 -1251500 -1787720 
+-1476381
-1737003 -1318660 -420744 -738887 -1025414 -952742 -1335695 -1441729 -
+1159485
-1171768 -1128103 -712072 -473309 -765466 -1198700 -1285089 -1348419 -
+1556346
-1327956 -971620 -1075261 -1120926 -1099457 -1471453 -1488643 -936739 
+-795910
-1015253 -1117698 -1274139 -1335401 -1178533 -1082915 -1199835 -137339
+8 -1226060
-1015291 -1051447 -997198 -1073021 -1428076 -1435493 -1098443 -1116945
+ -892701
174420 -83292 -1668146 -1877307 -1508544 -1280587 -504159 -607975 -142
+8293
-1301471 -664190 -733295 -1547665 -1889257 -1438016 -929650 -635668 -9
+66941
-1709318 -1526262 -652677 -626175 -1104370 -1161787 -1066086 -1021897 
+-1204104
-1547559 -1260868 -1043039 -1427884 -320452 516777 -1458465 -2573263 -
+1364426
-326671 -762368 -1766573 -1689927 -972438 -634770 -1046064 -1612223 -1
+623946
-1322425 -866368 -876067 -1189239 -934223 -881000 -1333817 -1397151 -1
+148026
-989331 -909200 -1025935 -1210654 -1219816 -1258397 -1316399 -1190640 
+-1113434
-1132030 -1068965 -1001816 -966472 -1267534 -1705439 -1276088 -680003 
+-888269
-1081254 -1002823 -970752 -977912 -1219760 -1427306 -1174881 -981218 -
+1224692
-1385454 -1194642 -1012963 -1104891 -1207310 -1220396 -1395342 -127909
+0 -845828
-1116560 -1437570 -1084730 -1159214 -1190535 -581785 -806557 -1415148 
+-1108899
-1013638 -1332982 -1215643 -1151227 -1200647 -968004 -899747 -1083941 
+-984857
-862614 -1092052 -1230587 -1167887 -1151473 -1140504 -1089388 -1087093
+ -1302907
-1332549 -1039985 -1436184 -1781060 -994629 -719614 -1290174 -1240472 
+-904403
-972206 -1244797 -1336973 -570295 -290696 -1693055 -2201096 -965058 -7
+70006
-1837310 -1719518 -770618 -361287 -586334 -1817225 -2347376 -837843 -2
+54105
-1492344 -1693600 -910120 -427102 -468579 -1605636 -2252801 -1104692 -
+140762
-848383 -1711137 -1131908 -630674 -1153093 -1475118 -1556041 -1417374 
+-833023
-760893 -1291887 -1430175 -1198517 -1146028 -1075614 -1099305 -1353092
+ -614445
-12355 -1355060 -2046745 -1349995 -1430142 -1206187 -98425 -430436 -16
+06793
-932603 -247395 -1634486 -1715870 -657143 -1626560 -2297483 -1002247 4
+62080
35877 -1586491 -1914975 -1532303 -1265764 -1183126 -1540281 -1654876 -
+932297
-85782 -1145971 -2049147 389008 879489 -2274618 -2809777 -1539767 -122
+2566
-880632 -1363944 -2055611 -1070531 -75843 -682017 -1416002 -1534265 -1
+508300
-1211030 -1108849 -1163247 -778277 -868413 -1526254 -1669297 -1274194 
+-798747
-752338 -1077686 -1162498 -1089816 -1196821 -1340222 -1368628 -1252791
+ -1054124
-1404053 -1889470 -1355151 -745665 -939884 -1186114 -1169937 -1105376 
+-1147347
-1181510 -890566 -773205 -1359124 -1502163 -796794 -906557 -1586946 -1
+377471
-1047865 -1030444 -1167082 -1588821 -1263972 -644461 -1008479 -1495011
+ -1342273
-1028058 -988989 -1158112 -1305765 -1315860 -1204892 -1050168 -848403 
+-1023304
-1482909 -1455809 -1284749 -1286722 -1159969 -1103699 -1106097 -943406
+ -952245
-1223480 -1299913 -1160612 -1080231 -1075582 -1058265 -1093522 -132459
+2 -1444696
-1195728 -1058915 -1149150 -1039824 -958094 -1193442 -1444084 -1485409
+ -1314866
-1219947 -1413455 -1335332 -837657 -819562 -1286799 -1379894 -1219435 
+-1124044
-1088228 -1260950 -1245597 -1092072 -1460263 -1665419 -1049773 -689335
+ -1060283
-1219379 -1033441 -975453 -966997 -1156753 -1512748 -1284994 -789371 -
+915644
-1169182 -1147150 -1197323 -1207674 -1067278 -933085 -912254 -1066721 
+-1266233
-1259568 -1045466 -1055480 -1359943 -1403499 -1186318 -1118476 -123297
+3 -1377352
-1290560 -1091953 -1136144 -1225847 -1210870 -1217571 -1231031 -118506
+9 -1113054
-1185330 -1240263 -1120751 -1083007 -993931 -883322 -1175803 -1493373 
+-1355860
-1063971 -1090329 -1345197 -1346019 -1172782 -1127428 -1075348 -110641
+7 -1380152
-1534778 -1463694 -1340902 -1313475 -1568310 -1514233 -954053 -873890 
+-1160319
-1197488 -1313393 -1451298 -1428033 -1021950 -166024 -271752 -1325705 
+-1737896
-1714508 -1495359 -934726 -727873 -838379 -733421 -736044 -1172448 -14
+24758
-1321563 -1314633 -1137709 -1170825 -1620191 -1250956 -635781 -1031522
+ -1500431
-1290616 -1021328 -1033316 -1074291 -1211802 -1334660 -1098954 -994158
+ -1316236
-1431913 -1191996 -1091457 -1165761 -1181486 -1213391 -1344006 -140999
+4 -1435200
-1465518 -1186385 -871016 -976628 -1179847 -1115625 -876088 -1004670 -
+1333060
-1284424 -1258914 -1180806 -833710 -904164 -1264856 -1219181 -1096460 
+-1234681
-1142026 -951652 -1160463 -1429389 -1354783 -1119268 -1141540 -1318320
+ -1315135
-1188595 -1141624 -1284897 -1377030 -1271011 -1023826 -862551 -918367 
+-949460
-1356037 -1786649 -1447122 -1153710 -1019925 -769824 -1136060 -1548436
+ -1111902
-818063 -1244032 -1361190 -1028930 -991652 -1018209 -1012953 -1295930 
+-1371684
-1180156 -1183918 -1240893 -1306134 -1325088 -1220611 -1324199 -134221
+6 -1023687
-872554 -913886 -957518 -1168996 -1446499 -1388738 -1089972 -930928 -9
+29609
-993027 -1015646 -1091791 -1305690 -1343565 -1165748 -1027871 -1012640
+ -1087786
-1186404 -1248425 -1198453 -1129382 -1193703 -1219190 -1148161 -120247
+3 -1261168
-1135200 -1121100 -1289463 -1250003 -1054067 -1120204 -1327277 -123378
+5 -1178537
-1604371 -1739003 -1218323 -1013978 -1188695 -1118849 -1110974 -112792
+3 -1056728
-1406357 -1491787 -959987 -905970 -1147927 -1064593 -1072575 -1156833 
+-1138653
-1216013 -1530899 -1760041 -1134627 -368485 -722795 -1403416 -1340010 
+-1021996
-1173518 -1490663 -1550328 -1375197 -1063157 -979212 -1067480 -1100436
+ -1256527
-1314402 -1182906 -1188332 -1242387 -1169014 -1093018 -1123957 -130452
+7 -1361532
-1192471 -1174671 -1136630 -973948 -987736 -1110964 -1264928 -1347238 
+-1307836
-1253470 -1170125 -1048140 -954800 -1021785 -1249522 -1450832 -1435361
+ -1167651
-980650 -1169147 -1170247 -820999 -1203242 -1873838 -1531581 -1050236 
+-1170669
-1117612 -940681 -1286462 -1517041 -1023648 -834009 -1215297 -1313855 
+-1338401
-1396640 -1132578 -950547 -1069690 -1148066 -1155598 -1273501 -1260452
+ -746425
-262995 -697223 -1568030 -1828812 -1702889 -902338 982211 1114728 -146
+2920
-2892609 -2117097 -1508292 -1451639 -1178200 -1153484 -1570561 -162302
+0 -1300785
-1068096 -1012333 -1261148 -1598204 -1564795 -1260201 -1236513 -133819
+6 -1107258
-936999 -986849 -1042388 -1261079 -1432733 -1208353 -1092080 -1253497 
+-1063788
-963919 -1329327 -1306502 -1044184 -1318947 -1537245 -1113093 -862395 
+-1103857
-1213260 -1183151 -1163644 -1117863 -1269569 -1515117 -1418782 -126965
+2 -1464783
-1353886 -843847 -815884 -1156185 -1418300 -1609264 -1370371 -910253 -
+1057106
-1470584 -1183571 -676737 -885380 -1272998 -1339584 -1383560 -1296692 
+-1141617
-1217674 -1179373 -903166 -896103 -1140247 -1177038 -1331049 -1919074 
+-1977233
-1138244 -516190 -773997 -1457803 -1489718 -935451 -903971 -1207930 -1
+307803
-1270151 -1159960 -1099448 -1090811 -1159112 -1266938 -1183558 -108122
+3 -997369
-1012252 -1529823 -1577220 -829769 -771159 -1241174 -1298038 -1143512 
+-1041211
-1042029 -1233145 -1466345 -1331412 -1146916 -1217168 -1108874 -104956
+7 -1149499
-1181035 -1363742 -1467882 -1281255 -1172830 -1020790 -858906 -1936347
+ -2879456
-1876257 -1128414 -810627 -210514 -434927 -585082 -472618 -1204344 -18
+93927
-1519070 -878879 -743883 -822486 -846772 -1018272 -1371794 -1550654 -1
+315731
-988149 -875586 -840765 -953618 -1241970 -1475921 -1401490 -1068513 -1
+151692
-1326206 -1016491 -1040386 -1406092 -1461914 -1335829 -1175231 -115881
+3 -1210189
-1037623 -919895 -1017856 -1110112 -1219289 -1346192 -1264686 -1107212
+ -1107887
-1086946 -1177693 -1436808 -1343801 -1201297 -1450223 -1395247
[download]

Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond

Comment on Re^4: Quick script to check data logger data Download Code

Replies are listed 'Best First'.
Re^5: Quick script to check data logger data by etj (Priest) on Aug 11, 2024 at 18:51 UTC
Now, the version that still uses Math::FFT, but PDL::Graphics::Simple only for visualisation (note the lack of needing to scale etc) - you need PDL, PDL::Graphics::Simple, and one underlying plotter, I recommend PDL::Graphics::Gnuplot: `use strict; use warnings; use Math::FFT; use PDL; use PDL::Graphics::Simple; my $filePath = $ARGV[0] // "testData.txt"; my @data = do { open my $fh, $filePath or die "Can't open $filePath: $ +!"; local $/; split /\s+/, <$fh> }; my $fft = Math::FFT->new(\@data); my $spectrum = $fft->spctrm; shift @$spectrum; # Remove DC signal component line pdl($spectrum); print "ret> "; <STDIN>;` [download] Even if you prefer other methods of calculation, I hope you agree that the quick visualisations available with P:G:S are a game-changer. (The last line is just so the plot window doesn't get immediately closed because your program terminated - that's not a problem if you use a REPL like `perldl`)	[reply] [d/l] [select]
Re^5: Quick script to check data logger data by etj (Priest) on Aug 11, 2024 at 20:55 UTC
And finally (cue human-interest story), the version using PDL::FFTW3; unfortunately for me I had to learn about power spectra and then convert Math::FFT's `spctrm` code into a small PDL function: `use strict; use warnings; use PDL; use PDL::FFTW3; use PDL::Graphics::Simple; my $filePath = $ARGV[0] // "testData.txt"; my @data = do { open my $fh, $filePath or die "Can't open $filePath: $ +!"; local $/; split /\s+/, <$fh> }; sub spectrum {my ($d)=@_; my $hn=$d->dim(0)-1; my $a2=$d->abs2/(($hn2)2); $a2->slice([1,$hn-1])=2; $a2; } my $spectrum = spectrum(rNfft1 pdl(\@data)); $spectrum = $spectrum->slice('1:-1'); # Remove DC signal component line pdl($spectrum); print "ret> "; <STDIN>;` [download] If you're into IRC, then next time you feel you might benefit from using PDL but don't feel 100% clear on where to start, do feel free to join #pdl on irc.perl.org and ask questions! A lightweight way to do so is on the MetaCPAN page for various PDL modules, with the "Chat with us" link.	[reply] [d/l] [select]
Re^5: Quick script to check data logger data by etj (Priest) on Aug 11, 2024 at 18:43 UTC
What? Software/hardware with bugs?? Once I'd emotionally recovered from dealing with that idea, I simplified your code into this (with your data above in a file you can give as an argument, or in the obvious default filename): use strict; use warnings; use Tk; use Tk::Canvas; use Math::FFT; my $filePath = $ARGV[0] // "testData.txt"; my @data = do { open my $fh, $filePath or die "Can't open $filePath: $ +!"; local $/; split /\s+/, <$fh> }; my $spectrum; my $fft = Math::FFT->new(\@data); $spectrum = $fft->spctrm; shift @$spectrum; # Remove DC signal component my $mw = MainWindow->new (-title => "Magnetometer Plotter"); my $canvas = $mw->Canvas (-height => 700, -width => 1024)->pack (); my $i = 0; $canvas->createLine( (map +(2 + 2 * $i++, $_), NormData($spectrum, 680)), -fill => 'blue' ); $mw->MainLoop; sub NormData { my ($data, $span) = @_; my ($min, $max); for my $datum (@$data) { $min //= $datum; $max //= $datum; $min = $datum if $min > $datum; $max = $datum if $max < $datum; } my $scale = $span / ($max - $min); map $span - ($_ - $min) * $scale + 10, @$data; } [download]	[reply] [d/l]
Re^5: Quick script to check data logger data by cavac (Prior) on Aug 12, 2024 at 17:00 UTC
There are bugs in the software that drive the logger and, after months of back and forth between us and the manufacturer, we haven't been able to get the system to work correctly. You wouldn't happen to have more details on that sensor setup? Maybe a datasheet, interface description, or anything of that sort? Maybe i could help to come with an alternative solution of getting the data out of the sensor. PerlMonks XP is useless? Not anymore: XPD - Do more with your PerlMonks XP Also check out my sisters artwork and my weekly webcomics	[reply]
Re^6: Quick script to check data logger data by GrandFather (Saint) on Aug 12, 2024 at 23:09 UTC
The sensor is just a solenoid about 2m long with, I'd guess, tens of thousands of turns. The electronics is a very sensitive DC coupled amplifier with extremely low low frequency (especially 1/f) noise followed by an ADC. The system is looking at low frequency changes in the local Earth's magnetic field due to changes in current flowing in the ionosphere which is influenced by the solar wind. A complication for our situation is that everything is running off solar power and there is no network connection available (cellular, POTS or fiber). The main issue we have with the logger is that it doesn't reliably recover from low power situations. It doesn't help that the system is powered over a 60m cable which generates about 1V drop between the 14V boosted supply and the logger. Things were much worse before I put a voltage boost system in at the supply end. Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond	[reply]
Re^7: Quick script to check data logger data by cavac (Prior) on Aug 14, 2024 at 06:27 UTC
At least for status monitoring (low bandwidth keep-alive packets), there are other options available. LORA (and/or LoraWAN) can bridge quite big distances line-of-sight. And there is the possibility of using something like an Arduino with a RockBLOCK Iridium sat module to send a daily "i'm still here" message that includes the battery charge status, if you can read that out from the solar charge controller. The Arduino also could monitor the logger itself and basically press the reset button if the logger has crashed. As for the power situation itself: The 60m cable is a big resistor and a decent amount of inductance as well, i suspect. If the logger pulls more power momentarily, the voltage will drop very sharply for a brief moment, then possibly rise above safe levels when power usage drops due to the inductance. The voltage instability problem will be possibly be worse if the power cable is buried in the soil, especially if it's wet, because on every change of power usage, this will change the voltage in the cable and the soil would act as the other end of a capacitor. Therefore, the voltage booster needs to go on the end of the cable where the logger is, with extra input capacitors. Making sure it can momentarily handle ovrevoltage as well and regulate that down. It should be a combined buck/boot converter. PerlMonks XP is useless? Not anymore: XPD - Do more with your PerlMonks XP Also check out my sisters artwork and my weekly webcomics	[reply]