Question vague so just random thoughts but . . .

• It's possible to use (say) Compress::Bzip2 to compress and uncompress strings in memory. You could certainly store a file of compressed data "lines" and read from that but the problem becomes knowing what's a "line" to read and then you're starting to need to implement your own file format with an index and . . . bleh. But not too bad (maybe inspired by the zip format have a TOC/index block at the end of the file which you read which provides a mapping of file offset (that you could pass to seek) and the number of bytes for that particular "line").
• Problem with that approach though is that by splitting things into separate lines compressed separately you're going to "lose" some of the redundancy that gzip was able to bite on.
• You haven't said why you need all these bits but something made me think of Bloom Filters; that may or may not be anywhere relevant. If you go a bit more into what you're trying to do with all of these you might get some more useful prior art pointers.

Update: Actually after thinking it over you might just could use Archive::Zip and let it store each of your "lines" as a separate file in the archive. That'd probably get you most of the way there without needing to reimplement any wheels. If you're truly worried about runtime space implement some sort of LRU cache on top of that and only keep the last n actively accessed "lines" live in memory.

Another thought depending on the sparseness of your bits using Set::IntSpan and a textual representation could have possibilities. But still . . . much handwavium here.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.