coldhawk has asked for the wisdom of the Perl Monks concerning the following question:

i was doing a program what uses flat database and i got this idea if i could compress my huge text file and then open that archive and then unpack some bytes(lines) in the memory so it would take less space and would be faster because the whole huge text file wouldn't be loaded in the memory, only some bytes(lines) of the file. so im asking you that anyone know any compressions that allow unpacking bytes from file in the archive not the whole file. and generally is this good idea in at all, what do you think people?

Replies are listed 'Best First'.
Re: read bytes from compressed file
by liz (Monsignor) on Nov 17, 2003 at 15:15 UTC
    If you're using Perl 5.8.0 or higher, you might want to try PerlIO::gzip.

    Otherwise I've personally always used an external unzipper in a pipe: 

    open (my $handle,"gunzip -c $filename |" ) or die $!;

    [download]

    Liz

[reply]
[d/l]
Re: read bytes from compressed file
by hardburn (Abbot) on Nov 17, 2003 at 15:32 UTC

    generally is this good idea in at all, what do you think people?

    At some point, you have to accept the fact that flat-files are a very inefficient database format, but yes, compression can help quite a bit. I added compression support to CGI::Search in version 0.5, and some simple benchmarks I ran against one of our real-world flat-file databases showed a 12-fold increase in speed. I don't think I have the benchmarking code around anymore, though, sorry. YMMV, especially since compression rates aren't guarenteed.

    ----
    I wanted to explore how Perl's closures can be manipulated, and ended up creating an object system by accident.
    -- Schemer

    : () { :|:& };:

    Note: All code is untested, unless otherwise stated

[reply]
[d/l]
Re: read bytes from compressed file
by zentara (Cardinal) on Nov 17, 2003 at 18:35 UTC
    The first thought that comes to my mind is make your own custom binary format. For instance maybe separate your flat database into A..Z and compress each section individually. Or maybe place some sort of "binary markers" in your compressed file, then search for it and only open that "chunk". Or... maybe keep a second index file, detailing where in the main file, certain sections begin and end, and just SEEK them out.

    You could get one of the compression modules off of cpan, and compress each record individually, then cat them together with a separator in between them. Basically you will end up with something like DBM.

[reply]

Back to Seekers of Perl Wisdom