Could 8 bit deltas for the x, y, z values gain you anything? If acceleration tends to be straight line then converting to a vector form may gain you something, especially if you then delta encode.

A quite different technique that works well for some types of data is to treat each bit position (or groups of adjacent bits - nibbles for example) in the data as a separate stream and run length encode each "stream". That gives excellent compression for the most significant bits and almost none for the least significant bits, but overall may give substantially better results than general purpose Huffaman type encoding.

Thanks for the suggestions! It took me a few days to think this through and try some things.

Indeed, per your suggestion I have tried deltas for x,y,z, although they don't all fit in 8 bits (some deltas will of course overflow 16-bit as well), so I use a variable width int (8, 16, or 32 bit), which was a pretty big win, even after compression.

The bitwise transposition you refer to is interesting, but was complex and didn't change things much once xz got ahold of it.

Then, you can divide the x,y,z deltas by the time deltas in order to get the derivate of the acceleration vector in order to get even more normalized values. The problem is that at this point you are using floats instead of integers, but unless you want absolute precision this should be a minor issue. Just select a resolution that fits your problem and discretize them accordingly.

Another possibility is to express the x,y,z deltas (or derivates) in the coordinate system defined by the Frenet-Serret frame. That would probably produce numbers that are nearer to zero and so, more compressible.

These suggestions were helpful. Thank you! Using deltas for x,y,z did help quite a bit.

The time delta derivative approach was interesting to think about, but would basically break the lossless compression I have going. At the moment, at least, I need to be able to reconstruct the original input verbatim, and the only way to do that is with fixed point arithmetic.

The one concession I was able to safely (verifiably) make is quantizing the time deltas that are close to the sensor frequency. The clock has a large (+/- 15%!) but predictable jitter pattern I was able to isolate and then basically RLE the result.

With the latest optimizations, I'm sitting at about 9:1 compression, which is acceptable. I was shooting for 10:1, but that last 33 KiB won't come easy.

Yes, but it performed slightly worse than xz, most likely because it uses the same compression algorithm.

-rw-r--r--  1   469K 10 Sep 19:33 accel_packed.7z
-rw-r--r--  1   559K 10 Sep 19:33 accel_packed.bz2
-rw-r--r--  1   559K 10 Sep 19:33 accel_packed.gz
-rw-r--r--  1   468K 10 Sep 19:33 accel_packed.xz
-rw-r--r--  1   560K 10 Sep 19:33 accel_packed.zip

# Uncompressed file, packed with Csss/CLsss:
-rw-r--r--  1   701K 10 Sep 18:34 accel_packed.dat
[download]

These are all about +/- 10%. Unless there's a significantly different and amazing general compression algorithm I haven't considered, I would imagine these results won't improve much, which is why I'm asking here to see if there's a way I can further optimize the data itself.

Would it help to convert your decimal values to a higher base, e.g., base 64? Consider the following:

use Modern::Perl;
use Math::BaseCnv;

my $original = <<END;
17077.161362,-595,2619,-15202
17077.174405,601,2115,-14859
17077.187416,-816,1894,-15539
17077.200402,210,1790,-15284
17077.213445,1560,1721,-15457
17077.226432,1408,1483,-15834
17077.239474,947,1161,-17531
17077.252517,689,1396,-17515
17077.265504,-861,1206,-18238
17077.278547,-493,1369,-17951
17077.291589,850,1294,-17360
17077.304576,1110,1215,-17271
17077.317619,326,1055,-13948
17077.330662,1980,1083,-16239
17077.343648,1295,1057,-16154
17077.356691,-1302,1207,-16277
17077.369734,-1986,1216,-16537
17077.382721,-134,1108,-17409
17077.395763,-739,1059,-16734
17077.408774,-2262,1006,-16690
17077.421761,-807,1174,-17046
17077.434803,-591,1081,-17615
17077.447846,-2080,-5757,-21463
17077.460833,-209,-17608,-19409
END

my @original = split ' ', $original;

say for @{ b10to64( \@original ) };

sub b10to64 {
    my $original = $_[0];
    chomp @$original;

    for ( my $i = 0 ; $i < @$original ; $i++ ) {
        my @entries = split ',', $$original[$i];
        my ( $whole, $decimal ) = split '\.', $entries[0];
        $entries[0] = cnv( $whole, 10, 64 ) . '.' . cnv( $decimal, 10,
+ 64 );
        $entries[$_] = cnv( $entries[$_], 10, 64 ) for 1 .. 3;
        $$original[$i] = ( join ',', @entries );
    }

    $original;
}
[download]

Output:

4Ar.dPI,-9J,ex,-3jY
4Ar.gb5,9P,X3,-3eB
4Ar.jmO,-Cm,Tc,-3op
4Ar.mxI,3I,R.,-3kq
4Ar.q75,OO,Qv,-3nX
4Ar.tI0,M0,NB,-3tQ
4Ar.wTo,Ep,I9,-4Hx
4Ar.zfb,An,Lq,-4Hh
4Ar.10qW,-DT,Is,-4S.
4Ar.140J,-7j,LP,-4OV
4Ar.17C5,DI,KE,-4FG
4Ar.1AN0,HM,I_,-4Dt
4Ar.1DYp,56,GV,-3Py
4Ar.1Gkc,Uy,Gx,-3zl
4Ar.1JvW,KF,GX,-3yQ
4Ar.1N5J,-KM,It,-3.L
4Ar.1QH6,-V2,J0,-42P
4Ar.1TS1,-26,HK,-4G1
4Ar.1Wdp,-BZ,GZ,-45U
4Ar.1Zp6,-ZM,Fk,-44o
4Ar.1c.1,-Cd,IM,-4AM
4Ar.1g9p,-9F,Gv,-4JF
4Ar.1jLc,-WW,-1Pz,-5FN
4Ar.1mWX,-3H,-4J8,-4lH
[download]

Conversion artifact:

4Ar.1c.1,-Cd,IM,-4AM
      ^
      |
      + - split on first '.' when converting back to base 10
[download]

This yielded a 30% reduction size of your data in accel.csv, which may help with your final data size after packing and compression.

Update: Evidently, this would not help...

The Data::BitStream and Data::BitStream::XS modules may be of some help (shameless plug since I'm the author). They're made for supporting the sort of compression you're looking at.

Using Adaptive Rice coding yields about 6.8:1, Exponential Golomb with best parameters about 6.9:1. xz/bzip2/gzip aren't able to help much with either result, yielding only about 7:1. So sadly not "significant" vs. 6.5:1.

One advantage of this is that, like your packing, it writes the values compressed, so no second stage of running xz needed.

There are lots of ways you could tweak the variable length output. Adaptive Rice works pretty well without a lot of thought about the parameters (the initial values used don't matter much as they'll adjust quickly). You could get complicated with Comma / Taboo, StartStop / StartStepStop, etc. codes if you wanted. It's also easy to do lossy coding by shifting the deltas before encoding and back again after decoding (taking care to keep symmetry in compressor/decompressor).

Quick example using your CSV:

#!/usr/bin/perl
use warnings;
use strict;
use Data::BitStream::XS;
use Text::CSV;
use autodie;
my $use_arice = 1;

my $csv = Text::CSV->new({ sep_char => ',' });

{
  my $file = 'accel.csv';
  open(my $data, '<', $file);
  my @prev = (0, 0, 0, 0);
  my @rice = (4, 4, 4, 4);
  my @egol = (8, 6, 6, 5);

  my $stream = Data::BitStream::XS->new( file => 'accel.gamma', mode =
+> 'w' );

  while (my $line = <$data>) {
    next if $line =~ /^;/;
    $line =~ s/[\r\n]//g;
    die "Bad CSV: $line" unless $csv->parse($line);
    my @fields = $csv->fields();
    $fields[0] = int($fields[0] * 10000 + 0.5);  # Drop last two digit
+s

    foreach my $n (0..3) {
      my $p = $prev[$n];
      my $v = $fields[$n];
      my $delta = $v - $p;
      my $udelta = ($n == 0) ? $delta
                             : ($delta >= 0)  ?  2*$delta  :  -2*$delt
+a-1;
      $use_arice ? $stream->put_arice($rice[$n], $udelta)
                 : $stream->put_expgolomb($egol[$n], $udelta);
    }
    @prev = @fields;
  }
  close($data);
  $stream->write_close;
}

# Verify we can read the stream back and write a CSV file
{
  my @prev = (0, 0, 0, 0);
  my @rice = (4, 4, 4, 4);
  my @egol = (8, 6, 6, 5);

  my $stream = Data::BitStream::XS->new( file => 'accel.gamma', mode =
+> 'ro' );
  my $file = 'new.csv';
  open(my $data, '>', $file);

  while (my $v = $use_arice ? $stream->get_arice($rice[0])
                            : $stream->get_expgolomb($egol[0])  ) {
    my $time = $prev[0] + $v;
    $prev[0] = $time;
    
    printf $data "%.4f", ($time/10000);

    foreach my $n (1..3) {
      my $udelta = $use_arice ? $stream->get_arice($rice[$n])
                              : $stream->get_expgolomb($egol[$n]);
      my $delta = (($udelta & 1) == 0) ? $udelta >> 1 : -(($udelta+1) 
+>> 1);
      my $v = $prev[$n] + $delta;
      print $data ',', $v;
      $prev[$n] = $v;
    }
    print $data "\n";
  }
  close($data);
}
[download]

Another idea I've seen work is to transpose your CSV, so that the first row is the timestamp, the second row is the X, etc. Then compress this. As the X's may be self-similar, and the Y's self-similar, but the X's and Y's are not, this may give the compression algorithm a break.

The timestamp should also be delta, except the first reading on resume.