rinceWind has asked for the wisdom of the Perl Monks concerning the following question:

Hello monks,

My client is a big user of the "compress" utility on Solaris 2.8; they use this in preference to gzip for some wierd reason.

The client has housekeeper scripts that compress log files nightly. I have a need to access the logs (uncompressed) from some perl scripts. I am looking for a module which understands this compression format.

I have tried Compress::Zlib and Compress:LZF, both in vain. Is there a module out there that will do the trick? The documentation on "compress" is scant to say the least: 

User Commands                                         compress(1)

NAME
     compress, uncompress, zcat - compress, uncompress  files  or
     display expanded files

SYNOPSIS
     compress [ -fv ]  [ -b bits ]  [ file ... ]

     compress [ -cfv ]  [ -b bits ]  [ file ]

     uncompress [ -cfv ]  [ file ... ]

     zcat [ file ... ]

DESCRIPTION
  compress
     The compress utility will attempt to reduce the size of  the
     named files by using adaptive Lempel-Ziv coding. Except when
     the output is to the standard  output,  each  file  will  be
     replaced  by  one  with  the extension .Z, while keeping the
     same ownership modes, change times and  modification  times.
     If  appending  the  .Z  to  the file pathname would make the
     pathname exceed 1023 bytes, the command  will  fail.  If  no
     files  are  specified, the standard input will be compressed
     to the standard output.

     The amount of compression obtained depends on  the  size  of
     the input, the number of bits per code, and the distribution
     of common substrings. Typically, text such as source code or
     English  is reduced by 50-60%. Compression is generally much
     better than that achieved by  Huffman  coding  (as  used  in
     pack(1)), and takes less time to compute. The bits parameter
     specified  during  compression   is   encoded   within   the
     compressed  file,  along  with a magic number to ensure that
     neither decompression of random data  nor  recompression  of
     compressed data is subsequently allowed.

  uncompress
     The uncompress utility will restore files to their  original
     state  after  they  have  been compressed using the compress
     utility. If no files are specified, the standard input  will
     be uncompressed to the standard output.

     This utility supports the uncompressing of  any  files  pro-
     duced  by  compress. For files produced by compress on other
     systems, uncompress supports 9- to 16-bit  compression  (see
     -b).

  zcat
     The  zcat  utility  will  write  to  standard   output   the
     uncompressed  form  of files that have been compressed using
     compress. It is the equivalent of uncompress -c. Input files
     are not affected.

SunOS 5.8            Last change: 9 Sep 1999                    1

User Commands                                         compress(1)

OPTIONS
     The following options are supported:

          -c    Write  to  the  standard  output;  no  files  are
                changed and no .Z files are created. The behavior
                of zcat is identical to that of `uncompress -c'.

          -f    When compressing, force compression of file, even
                if  it  does  not actually reduce the size of the
                file, or if the corresponding file.Z file already
                exists.  If  the  -f option is not given, and the
                process is not running in the background,  prompt
                to  verify whether an existing file.Z file should
                be overwritten. When uncompressing, do not prompt
                for  overwriting  files.  If the -f option is not
                given, and the process  is  not  running  in  the
                background,  prompt to verify whether an existing
                file should be overwritten. If the standard input
                is  not  a  terminal and -f is not given, write a
                diagnostic message to  standard  error  and  exit
                with a status greater than 0.

          -v    Verbose. Write to standard  error  messages  con-
                cerning  the percentage reduction or expansion of
                each file.

          -b bits
                Set the upper limit (in  bits)  for  common  sub-
                string  codes.  bits must be between 9 and 16 (16
                is the default). Lowering the number of bits will
                result in larger, less compressed files.

OPERANDS
     The following operands are supported:

          file  A path name of a file to be compressed.  If  file
                is  -,  or  if no file is specified, the standard
                input will be used.

USAGE
     See largefile(5) for the  description  of  the  behavior  of
     compress,  uncompress,  and  zcat  when  encountering  files
     greater than or equal to 2 Gbyte ( 2**31 bytes).

ENVIRONMENT VARIABLES
     See environ(5) for descriptions of the following environment
     variables that affect the execution of compress, uncompress,
     and zcat: LC_CTYPE,  LC_MESSAGES,  and NLSPATH.

SunOS 5.8            Last change: 9 Sep 1999                    2

User Commands                                         compress(1)

EXIT STATUS
     The following error values are returned:

     0     Successful completion.

     1     An error occurred.

     2     One or more files were  not  compressed  because  they
           would  have  increased  in size (and the -f option was
           not specified).

     >2    An error occurred.

ATTRIBUTES
     See attributes(5) for descriptions of the  following  attri-
     butes:

     ____________________________________________________________
    |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
    |_____________________________|_____________________________|
    | Availability                | SUNWesu                     |
    |_____________________________|_____________________________|
    | CSI                         | Enabled                     |
    |_____________________________|_____________________________|

SEE ALSO
     ln(1), pack(1), attributes(5), environ(5), largefile(5)

DIAGNOSTICS
     Usage: compress -fvc -b maxbits file... 
           Invalid options were specified on the command line.

     Missing maxbits
           Maxbits must follow -b,  or  invalid  maxbits,  not  a
           numeric value.

     file: not in compressed format
           The  file  specified  to  uncompress  has   not   been
           compressed.

     file: compressed with xxbits, can only handle yybits
           file was compressed by a program that could deal  with
           more  bits  than  the  compress  code on this machine.
           Recompress the file with smaller bits.

     file: already has .Z suffix -- no change
           The file is assumed to be already  compressed.  Rename
           the file and try again.

     file: already exists; do you wish to overwrite (y or n)?
           Respond y if you want the output file to be  replaced;

SunOS 5.8            Last change: 9 Sep 1999                    3

User Commands                                         compress(1)

           n if not.

     uncompress: corrupt input
           A SIGSEGV violation was detected, which usually  means
           that the input file is corrupted.

     Compression:  xx.xx%
           Percentage  of  the  input   saved   by   compression.
           (Relevant only for -v.)

     - - not a regular file: unchanged
           When the input file is not a regular file, (such as  a
           directory), it is left unaltered.

     - - has xx other links: unchanged
           The input file has links; it is left  unchanged.   See
           ln(1) for more information.

     - - file unchanged
           No savings are  achieved  by  compression.  The  input
           remains uncompressed.

     filename too long to tack on .Z
           The path name is too long to append the .Z suffix.

NOTES
     Although compressed files are  compatible  between  machines
     with large memory, -b 12 should be used for file transfer to
     architectures with a  small  process  data  space  (64KB  or
     less).

     compress should be more flexible about the existence of  the
     .Z suffix.

--
I'm Not Just Another Perl Hacker

  • Comment on Wanted: perl equivalent of Solaris compress utility

Replies are listed 'Best First'.
Re: Wanted: perl equivalent of Solaris compress utility
by bart (Canon) on Nov 16, 2004 at 13:51 UTC
    Your quest reminds me of something I read in the FAQ for Zlib, (Zlib FAQ), question 12:
    Can zlib handle .Z files?
    No, sorry. You have to spawn an uncompress or gunzip subprocess, or adapt the code of uncompress on your own.

    Note: Zlib is the basis for the module Compress::Zlib, and the basis for the gzip utility. The reason is not that these people can't code it, but for legal reasons: they can't distribute the code to do it — or at least, it'd compromoise the rest of their library. I think details of that decision are somewhere on that site. (So far, my searches for it have been unsuccessful.)

    Therefore, you mustn't expect it, or likely any other perl library, to support .Z files, at least, until the patent(s) .Z is under, expire.

    Update According to this site, which describes the hisory a little, the patent should already have expired in 2001.

[reply]
      I've heard the patent problem is trickier than that. The patent has expired in the US but not everywhere else.
[reply]
Re: Wanted: perl equivalent of Solaris compress utility
by muntfish (Chaplain) on Nov 16, 2004 at 11:53 UTC

    It seems odd that there is no Perl module to handle .Z files.

    But if this is the case then you have (at least) two options:

    • uncompress the file to a temporary directory (using system() or backticks) then open it as normal.
    • read the output of zcat: open LOG, "zcat $zfile |" or die "...";

    If disk space is at a premium then the second option may be the better one. Hope this helps.

    s^^unp(;75N=&9I<V@`ack(u,^;s|\(.+\`|"$`$'\"$&\"\)"|ee;/m.+h/&&print$&
[reply]
[d/l]
[select]
Re: Wanted: perl equivalent of Solaris compress utility
by PodMaster (Abbot) on Nov 16, 2004 at 12:19 UTC
[reply]
      Thanks for the pointers, especially LZW Demystified. I've tried Compress::LZW, and as the documentation says, it does not yet support compress(1)'s .Z files.

      I've also tried compressing a small file and attempting to decode it by hand with no success. Here's a challenge: could someone show how the algorithm decompresses the following: 

      /home/test: od -bc foo.Z
0000000 037 235 220 124 320 224 001 321 346 315 033 067 040 322 314 041
         037 235 220   T 320 224 001 321 346 315 033   7     322 314   !
0000020 030 206 314 300 067 146 100 214 021 130 146 116 031 005
         030 206 314 300   7   f   @ 214 021   X   f   N 031 005
0000036
/home/test:

      --
      I'm Not Just Another Perl Hacker

[reply]
Re: Wanted: perl equivalent of Solaris compress utility
by TedPride (Priest) on Nov 16, 2004 at 12:44 UTC
    If you already have an uncompress utility installed, no point rolling your own. I'd personally do what muntfish suggests and call the system to uncompress the file, then process the result.
[reply]
Re: Wanted: perl equivalent of Solaris compress utility
by Old_Gray_Bear (Bishop) on Nov 16, 2004 at 16:25 UTC
    I just saw in Google-News headlines that Solaris 10 was resleased as Open Source. You might have a chat with your local Sun Micro reps and see if thay have a problem with you reading the code to compress(1) and proting/adapting/creating a Perl module incorporating the compress(1) algorithm.

    ----
    I Go Back to Sleep, Now.

    OGB

[reply]
Re: Wanted: perl equivalent of Solaris compress utility
by elwarren (Priest) on Nov 16, 2004 at 16:57 UTC
    ...and because gzip was not bundled with the OS until 2.8, IIRC. They've only recently started bundling seemingly neccessary utilities like top, ssh, and bash. I think top was included in Solaris 9?

    Basically, when I need to restore my system from backups after a crash, I can be sure that compress is there, but not sure about gzip.
[reply]
Re: Wanted: perl equivalent of Solaris compress utility
by bluto (Curate) on Nov 16, 2004 at 16:29 UTC
    My client is a big user of the "compress" utility on Solaris 2.8; they use this in preference to gzip for some wierd reason.

    FWIW, in the past there were several reasons for using compress instead of gzip, but these have pretty much evaporated over time: it was even more ubiquitous than gzip; it can be much faster for certain highly compressible files; I vaguely remember that at one point gzip had problems with large files.

[reply]

Back to Seekers of Perl Wisdom