My advice, since it's a huge secured file, is to run the /usr/local/bin/gpg binary from Perl, using system, qx, backticks, IPC3 or whatver method you choose. The c program will do it right and faster than perl on a huge file, and you won't have to worry about huge memory gains which come in a Perl program after using alot of memory. For example see: perl gpg system calls

Using GnuPG already calls /usr/local/bin/gpg externally, you can see it running as a separate process

#!/opt/perl/bin/perl -w

#use strict;
use GnuPG qw( :algo );

my $gpg = GnuPG->new(gnupg_path => "/usr/local/bin/gpg", homedir => "/
+home/rolivier/.gnupg");

my $encfile='/home/ro/big.testfile.gpg';
$outfile = $encfile;
$outfile =~ s/.gpg//;
my $pp='Passphrase';

$gpg->decrypt(ciphertext => $encfile, output => $outfile, passphrase =
+> $pp);
[download]

I'm thinking of using Crypt::OpenPGP for small files < 1MB and the GnuPG for the rest. The program I'm writing will be processing thousands of small files, so repeatedly calling an external process isn't the best solution. That's why I was looking for a pure-perl implementation.

The program I'm writing will be processing thousands of small files, so repeatedly calling an external process isn't the best solution. That's why I was looking for a pure-perl implementation.

If the module already calls /usr/local/bin/gpg externally, then you have a different problem. You need to be sure that you are not creating a new Crypt::OpenPGP object for each file, otherwise you will be causing alot of overhead. What you want to do is create 1 Crypt::OpenPGP object and reuse it. You must clear out all the old data in the object after each use. See

7.27: How do I clear a package?

    Use this code, provided by Mark-Jason Dominus:

        sub scrub_package {
            no strict 'refs';
            my $pack = shift;
            die "Shouldn't delete main package"
                if $pack eq "" || $pack eq "main";
            my $stash = *{$pack . '::'}{HASH};
            my $name;
            foreach $name (keys %$stash) {
                my $fullname = $pack . '::' . $name;
                # Get rid of everything with that name.
                undef $$fullname;
                undef @$fullname;
                undef %$fullname;
                undef &$fullname;
                undef *$fullname;
            }
        }

    Or, if you're using a recent release of Perl, you can just use the
    Symbol::delete_package() function instead.
[download]

So now you can reuse that same opening of /usr/local/bin/gpg, which will speed things up and reduce memory usage. You would still have 1 memory problem however, because the perl object's memory usage will swell to the largest file size it encounters.

So you may have to filter your files for size, to decide to feed them directly to the binary, or use the reusable module.

There is also the possibility of running parallel processes in threads if you have a multicore computer. But that is a whole different thread on it's own, running gpg thru threads.

1 last bit of advice, run all your decoding on a ramdisk if it is to be kept private. :-)

---

I'm not really a human, but I play one on earth. ..... an animated JAPH

Hello roperl,

Welcome to the monastery. I have never used the Crypt::OpenPGP but by reading the documentation why don't you the encrypted line one by one in the file and decrypt it?

Sample of code from similar question Crypt::OpenPGP, encrypting messages using a public key file:

$pgp = new Crypt::OpenPGP( SecRing => $private_file );
my $plaintext = $pgp->decrypt (
                               Data => $cyphertext,
                               Passphrase => $pass,
                              ) || die $pgp->errstr();
[download]

Hope this helps, BR.

The input file a mostly in binary format, so processing 1 line at a time isn't possible

Hello again roperl,

I do not know if you have managed to resolve your problem but I was trying to search through the web regarding your question and I found this article that I think is useful The problem of decrypting big files encrypted with openssl smime.

It is not referring to OpenPGP but more to encryption limitations I assume are applied the same problems. From the link above:

Yes you will hit that problem with large files. The S/MIME encrypt str
+eams with a low memory overhead while decrypt does not stream and nee
+ds the whole 
file in memory. It's a serious amount of effort to stream decrypt and 
+so far there hasn't been enough interest to do that.
[download]

Reading it maybe it will assist you or other monks to have a better understanding. It is not a solution to your problem but more about general knowledge.

Hope this helps, BR.

Is there anything wrong with that? Need to open $fh to output file and then print to the output file the plain text after it was decrypted. Still the issue is the decrypt function returns to variable as opposed to being able to pass output filehandle

Well, you could just as well do this:

$fh = $pgp->decrypt(Filename => $file, Passphrase => $pp);
[download]

But depending on how that method works, you may not see any improvement. Worth a try though.

Is there anything wrong with that? Need to open $fh to output file and then print to the output file the plain text after it was decrypted.

But your code does not try to print anything to $fh, all it does is create an empty file ..

Still the issue is the decrypt function returns to variable as opposed to being able to pass output filehandle

Yeah decrypt doesn't do that

You can make a feature request or bug report at rt://Crypt-OpenPGP ( bug-Crypt-OpenPGP@rt.cpan.org )

It would be nice if decrypt didnt slurp giant files