Unfortunatelly 'Industrial strength archiving' means non-perl, (well, could be perl if someone would sit and create some libs and utils...)

Why? Because heavyweigth archiving means bypassing(at least partly) filesystem structure (i.e. - not "open /, list all dirs, open every dir, list all files, archive them... etc...", but: linearly walk bytestream and archive what's marked for archiving ).

At this point in time only tools from dump family can do tricks like that. I use mostly xfs, and xfsdump fits the bill, and beats all perl-lib-accessible methods like Archive::Tar by so wide margin that they're not really in the same competition. ( and on heavilly loaded system this means - xfsdump finishes dump, and tools like tar produce only unusable garbage ).

Now, for compression...lzop is great tool, I find it extremely usefull (it offers very fast compression with compression ratios hovering around what gzip -1 achieves). And it can compress streams, so you shouldn't have any trouble with piping to it and from it from perl.

Another toy, that I found very neat and usefull in Archiving business is rzip. This is definitelly not industrial strength type of solution, because it's very young ( and for your use, it cannot and will not support compressing streams... ), but it easily outperforms bzip2 -9 by a healthy 10-30%.

This is very significant achievement, what is surprising, is that while working on typical backup archives (multi-gigabyte files) it works sometimes several times faster then bzip2, while still outperforming it on compression ratio front.

Of course you need rather healthy machine to run it, because it's working set hovers around 0.5G...

In reply to Re: Industrial strength archiving by Eyck
in thread Industrial strength archiving by rinceWind

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.