I have a large array that contains sets of integers.

array[0]="1,100,105,1003";
array[1]="42,314,2718,30302";
etc...
[download]

I would like to save RAM with a more efficient representation of these sets. Here are a few facts about the numbers:

• The numbers do not repeat
• The numbers represent an index into another array
• I can reorder the data if I really need to
• Some numbers are much more common than others
• I don't care much about how long it takes to compress the data
• I would like to decompress the data relatively quickly
• In some cases there are thousands of entries per row, other cases only one.

The best answer might be to just use a database. I was thinking about a RAM-based solution in an attempt to get blazing speed in a mod_perl application. It would be nice if I could look up and decode about 100 of these rows in 0.05 seconds or so.

I would also like to be able to load these numbers into RAM relatively quickly, so I would like a data structure that can be stored and restored with Storable or some other quick method.

I am looking for ideas, pointers to algorithms, modules, or code. It seems like a fairly fundamental area of computer science, but I don't know what this area is called.

Thanks!

It should work perfectly the first time! - toma

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.