in reply to Using Crypt::OpenPGP with large files

I can do this with GnuPG, but it calls an external gpg program binary ie "/usr/local/bin/gpg" ..... The Crytp:OpenPGP module does this with a pure perl implementation However, the decrypt function returns the decrypted file into a scalar variable

My advice, since it's a huge secured file, is to run the /usr/local/bin/gpg binary from Perl, using system, qx, backticks, IPC3 or whatver method you choose. The c program will do it right and faster than perl on a huge file, and you won't have to worry about huge memory gains which come in a Perl program after using alot of memory. For example see: perl gpg system calls


I'm not really a human, but I play one on earth. ..... an animated JAPH
  • Comment on Re: Using Crypt::OpenPGP with large files

Replies are listed 'Best First'.
Re^2: Using Crypt::OpenPGP with large files
by roperl (Beadle) on Jul 25, 2017 at 14:26 UTC
    Using GnuPG already calls /usr/local/bin/gpg externally, you can see it running as a separate process
    #!/opt/perl/bin/perl -w #use strict; use GnuPG qw( :algo ); my $gpg = GnuPG->new(gnupg_path => "/usr/local/bin/gpg", homedir => "/ +home/rolivier/.gnupg"); my $encfile='/home/ro/big.testfile.gpg'; $outfile = $encfile; $outfile =~ s/.gpg//; my $pp='Passphrase'; $gpg->decrypt(ciphertext => $encfile, output => $outfile, passphrase = +> $pp);
    I'm thinking of using Crypt::OpenPGP for small files < 1MB and the GnuPG for the rest. The program I'm writing will be processing thousands of small files, so repeatedly calling an external process isn't the best solution. That's why I was looking for a pure-perl implementation.
      The program I'm writing will be processing thousands of small files, so repeatedly calling an external process isn't the best solution. That's why I was looking for a pure-perl implementation.

      If the module already calls /usr/local/bin/gpg externally, then you have a different problem. You need to be sure that you are not creating a new Crypt::OpenPGP object for each file, otherwise you will be causing alot of overhead. What you want to do is create 1 Crypt::OpenPGP object and reuse it. You must clear out all the old data in the object after each use. See

      7.27: How do I clear a package? Use this code, provided by Mark-Jason Dominus: sub scrub_package { no strict 'refs'; my $pack = shift; die "Shouldn't delete main package" if $pack eq "" || $pack eq "main"; my $stash = *{$pack . '::'}{HASH}; my $name; foreach $name (keys %$stash) { my $fullname = $pack . '::' . $name; # Get rid of everything with that name. undef $$fullname; undef @$fullname; undef %$fullname; undef &$fullname; undef *$fullname; } } Or, if you're using a recent release of Perl, you can just use the Symbol::delete_package() function instead.
      So now you can reuse that same opening of /usr/local/bin/gpg, which will speed things up and reduce memory usage. You would still have 1 memory problem however, because the perl object's memory usage will swell to the largest file size it encounters.

      So you may have to filter your files for size, to decide to feed them directly to the binary, or use the reusable module.

      There is also the possibility of running parallel processes in threads if you have a multicore computer. But that is a whole different thread on it's own, running gpg thru threads.

      1 last bit of advice, run all your decoding on a ramdisk if it is to be kept private. :-)


      I'm not really a human, but I play one on earth. ..... an animated JAPH