Doing it in memory is by far the easiest thing. It is also simple with one line calls to decrypt/encrypt and compress/decompress. This uses Crypt::Blowfish but you could use your favourite block cipher. The order is VITAL. plaintext->compress->crypt->file is the only way to go (reverse on the way out of course).

By definition a decent encryption algorithm will turn '0'x1000000 into random (and thus totally uncompressable noise). On the other hand a compression algorithm will compress the hell out of the plaintext due to the repeated pattern. Then you can crypt that. For interest reverse the order of crypt<->compress and you will see no compression at all. This is a simple measure of the quality of the encryption - the compression algorithm can no longer see any patterns (and we know the plaintext has patterns). Crap encryption WILL compress. Try ROT13, or bongo crypt ;-)

$|++;
use Compress::Zlib ;
use Crypt::CBC;
my $cipher = Crypt::CBC->new( {'key' => 'secret key', 'cipher' => 'Blo
+wfish',  });
my $file = 'c:/text.txt';

write_file( $file, $cipher, "abcbefghijklmnopqrstuvwxyz\n"x1000 );
print "\n\nDecrypting and decompressing\n\n";
my $data = read_file( $file, $cipher );
printf "Got back %d bytes\n%s\n[snip]", length($data), substr($data,0,
+64);

sub write_file {
    my ( $file, $cipher, $data ) = @_;
    printf "Before compression %d bytes\n", length($data);
    $data = compress($data);
    printf "After compression %d bytes\n", length($data);
    $data = $cipher->encrypt($data);
    printf "After encryption %d bytes\n", length($data);
    open my $fh, '>', $file or die $!;
    print $fh $data;
    close $fh;
    print map{ s/[^\040-\177]//g;$_ }$data;
}

sub read_file {
    my ( $file, $cipher ) = @_;
    open my $fh, $file or die $!;
    local $/;
    my $data = <$fh>;
    close $fh;
    $data = $cipher->decrypt($data);
    $data = uncompress($data);
  return $data;
}
__DATA__
Before compression 27000 bytes
After compression 116 bytes
After encryption 136 bytes
RandomIV#vTP;\\-Ij{Et@t--sC7SWy{<6P~)}y9p6_r$0bB(d1$k8:6,

Decrypting and decompressing

Got back 27000 bytes
abcbefghijklmnopqrstuvwxyz
abcbefghijklmnopqrstuvwxyz
abcbefghij
[snip]
[download]

By definition a decent encryption algorithm will turn '0'x1000000 into random (and thus totally uncompressable noise).
...
This is a simple measure of the quality of the encryption - the compression algorithm can no longer see any patterns (and we know the plaintext has patterns).

I don't think this is true of one time pads. I can imagine a situation where a OTP could produce output that could then be highly compressed. Of course, this is a special case nitpick :).

What on earth makes you think data encrypted with a one time pad contains redundant data repeats and can thus be compressed? Please see Elgons note at Re^2: Compressing and Encrypting files on Windows and follow the link.

• Crypt::CBC The encryption and decryption process is about a tenth the speed of the equivalent SSLeay programs (compiled C). This could be improved by implementing this module in C. It may also be worthwhile to optimize the DES and IDEA block algorithms further.
• Crypt::Rijndael_PP (AES)
• Crypt::Blowfish_PP

Thanks for the input.

I am afraid you use of the phrase "Pure Perl" is confusing me. Normally I would take this to mean a module that is implemented purely in Perl (without any calls to C etc), but that is not the way you are using it (since at least the CPAN implementations of Blowfish and Rijndael I am using have binary elements).

I guess when you say "Pure Perl" you are focusing on the fact that these can be obtained from CPAN (and / or ActiveState) rather than the other challange of implementing modules on OSs that provide an inadequate set of tools for developers (where if perl is available but there is no good freely available C compiler a Pure Perl module will always work).

Please don't take this as sarcasm, but the very first line in the POD for the modules says:

Blowfish encryption algorithm implemented purely in Perl.

There are no calls to C and these can be used in place of the existing modules.

My suggestion had nothing to do with OS of choice. The PP modules will allow you to run the encryption code inline with your existing Perl and no forking or external process calls will be needed. Depending on how many calls you're making, this may be faster since you can avoid the context switch and overhead of starting an extproc.

This is why I referenced my POE example. POE doesn't like external programs because of it's own internal event loop signal handling. Using the PP modules allows me to avoid this.

Glad you got your code working. I'd be interested in what kind of performance you get, as I've been thinking about implementing something similar.

The way I solved it

Thanks to all those of you who replied to my post. I thought it would be worthwhile to hint at how I solved the issue in case someone has a similar problem later. I mentioned in the post I am using various tools (like XML::SAX) that require file handles of some kind, so any solution has to involve producing IO::Handle like objects.

When reading data this is no problem, I can just read all my file into a scalar, use my encryption algorithm to decrypt then inflate and finally use the reulting vector to create an IO::Handle via IO::Scalar.

When writing data the issue is more complex, I have to pass something that the writing modules can treat as a file handle, but when they are done make sure that the post processing happens. I could have changed my code to add an extra "now save this file" step after every place where files are written, but I decided to create my own IO::Thing to do that work for me.

My first thought was to have an object that held a handle and some extra info, pass all calls through to the handle except close() and when the object is closed do the extra work. That doesn't work because handles are not objects (and life is too short to understand as much about them as I want to).

My second thought was to create a IO::Scalar subclass that can perform a special action when the close it called, that doesn't work either because IO::Scalar implements its objects as a localised GLOB.

My third thought was to badly mangle the definition of IO::Scalar to force the behaviour I wanted. This I did creating my own class called IO::ActionOnClose (the implementation is so horrible I won't troble you with it).

With this class my open function now looks like:

sub _open_file
  {
    my($file_name,$mode) = @_;

    $mode = "r" if(!defined($mode));

    my $compress = 1;
    $compress = "" if($file_name =~ /\.xml$/i);

    if($mode eq "r")
      {
        my $fh = new IO::File($file_name);
        binmode $fh;

        my $f1 = "";

        while(!$fh->eof())
          {
            my $c = $fh->read($f1,1024*16,length($f1));
          }
        $fh = undef;

        my $f2 = "";
        if(defined($model_passphrase) && $model_passphrase ne "")
          {
            # Pad $f1 to the next 8 byte boundary
            if((length($f1) % 8) != 0)
              {
                $f1 .= "\x00" x (8 - (length($f1) % 8));
              }

            my $cipher =  new Crypt::Blowfish $model_passphrase;
            for(my $i=0;(8*$i)<length($f1);$i++)
              {
                $f2 .= $cipher->decrypt(substr($f1,8*$i,8));
              }
            $f2 =~ s/\x00+$//s;
          }
        else
          {
            $f2 = $f1;
          }

        # Just in case the file is big save some memory
        $f1 = "";

        my $f3 = "";
        if($compress)
          {
            $f3 = uncompress($f2);
          }
        else
          {
            $f3 = $f2;
          }
        return new IO::Scalar \$f3;
      }
    else
      {
        # When writing I want to first get the output, then compress
        # then encrypt, then write to the file.

        # So I have to create a handle that does some magic 
        # when it is closed
        my $buffer;
        return new IO::ActionOnClose(\$buffer,
                         action => \&_send_file,
                         args => [\$buffer,$file_name,$compress,$model
+_passphrase]);
      }
  }

}

sub _send_file
  {
    my($f3_ref,$file_name,$compress,$model_passphrase) = @_;

    # This is the reverse of what _open_file does for
    # read (even the variable names are the same)
    my $f2;
    if($compress)
      {
        $f2 = compress(${$f3_ref});
      }
    else
      {
        $f2 = ${$f3_ref};
      }
    my $f1 = "";
    if(defined($model_passphrase) && $model_passphrase ne "")
      {
        my $cipher =  new Crypt::Blowfish $model_passphrase;

        # Pad $f1 to the next 8 byte boundary
        if((length($f2) % 8) != 0)
          {
            $f2 .= "\x00" x (8 - (length($f2) % 8));
          }

        for(my $i=0;8*$i<length($f2);$i++)
          {
            $f1 .= $cipher->encrypt(substr($f2,8*$i,8));
          }
      }
    else
      {
        $f1 = $f2;
      }

    my $fh = new IO::File($file_name,"w");
    binmode $fh;
    $fh->syswrite($f1,length($f1));
    $fh->close();
  }
[download]

Once again thaks to those who contributed and I hope this helps someone else

That should be the way it works from CLI. Not all at the same time.