Just storing your number in binary rather than ascii with delimiters would cut your memory requirements to about 1/3 on average. Assuming that your example is representative and your numbers will fit in 16-bits. You mentioned the figures of 50,000 sets and assuming 4 per set as shown.

Instead of approx 1 MB, your down to 390k, ignoring the cost of the arrays which should be constant That's a start.

Projecting the timing of the code below gives around 3 seconds to pack the 50k x 4 x 16 bits and 0.002 secs to unpack 100 lines, which comfortably fits your spec, and my machine is hardly the quickest around. 

#! perl -slw
use strict;
use Benchmark::Timer;
my $t= Benchmark::Timer->new();


my (@ary1, @ary2);

#! 1000 lines of 4 random numbers (0-65536) in ascii with commas.
@ary1 = map{ join ',', map{ int rand(2**16) } 1..4 } 1 .. 1000;
print $ary1[0];

$t->start('packing');
#! The same numbers stored as binary strings
@ary2 = map{ pack 'S4', split',', $_ } @ary1;
$t->stop('packing');
print length $ary2[0];

print '1000 in ascii:  ', scalar @ary1, ' : ', length "@ary1";
print '1000 in binary: ', scalar @ary2, ' : ', length "@ary2";

$t->start('unpacking');
for (@ary2) {
    my @line_of_nums = unpack 'S4', $_;
#    print "@line_of_nums";
}
$t->stop('unpacking');

$t->report();
__END__
C:\test>229792
52318,4206,9238,48758
8
1000 in ascii:  1000 : 22310
1000 in binary: 1000 : 8000
1 trial of packing (50ms total)

1 trial of unpacking (20ms total)

[download]

Examine what is said, not who speaks.

The 7th Rule of perl club is -- pearl clubs are easily damaged. Use a diamond club instead.

In reply to Re: Compressing a set of integers by BrowserUk
in thread Compressing a set of integers by toma

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.