Many thanks for all the responses, especially I0 for the brilliant snippet of code and bart for the clear explanation of the benefits of pack with the "w" template. All of the responses were valuable; they really improved my understanding. I have implemented a solution along these lines and now I only need about 26% as much memory as I did with the ascii dataset. I am delighted with this solution!

With this solution, I expect to save about 1.5 million numbers in about 2 megabytes of storage. Wow!

Here is the proof of concept code that I ended up with:

use Storable;
use File::Flat;

my @to_save;
my $nrows=10000;
my $rowmax=300;

for (0..$nrows) {
  my @r;
  $r[0]=int(rand(20));
  for (1..int(rand($rowmax))) {
    push @r, $r[$_-1] + 1 + int(rand(200));
  }
  push @to_save, join ',',@r;
}

my $in="";
foreach( @to_save ) {
  $in .= $_."\n";
  my @a = sort {$a<=>$b} split/,/;
  my $d=0;
  for( @a ){ ($d,$_)=($_,$_-$d); }
  $_ = pack'w*',@a;
}

open(TEXT, ">array.txt") or die "Can't open output file";
print TEXT $in;
close TEXT;

store (\@to_save, 'array.sto');

my $to_retrieve= retrieve ('array.sto');
my $out="";
foreach( @$to_retrieve ) {
  my @a = unpack'w*',$_;
  my $c = 0;
  foreach my $n (@a) {
    $c += $n;
    $out .= $c.",";
  }
  chop $out;
  $out .= "\n";
}

if ($in eq $out) {
  print "OK\n"
}

my $sz_in= File::Flat->fileSize('array.txt')
             or die "Can't size array.txt";
my $sz_out= File::Flat->fileSize('array.sto')
             or die "Can't size array.sto";

print "Final size = ",int(100*$sz_out/$sz_in+0.5),"% of input\n";

[download]
It should work perfectly the first time! - toma

In reply to Compressing a set of integers - Solution by toma
in thread Compressing a set of integers by toma

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.