Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hi there Monks!
I am looking all over for something like this code, but I still can't figure this one out, I need to zip files inside directories but the zip file can not be larger than 5MB each, but all the files inside of these directories will be more than that, I need to find a way to create multiple .zips files in case I read directories and the sum of files is greater than 5MB zipped. This code is almost there but I am stuck on the logic of making multiple zips if necessary.
Here is the code and thanks for the help!
#!/usr/bin/perl -w

use strict;
use Archive::Zip qw/AZ_OK/;
use File::Temp qw/tempfile/;

use constant MB => 1024 * 1024;

my $dir = qw( zip_this/);
my @files = do {
 opendir my $fd, "$dir" or die $! or die $!;
 grep -f, map  "$dir$_", readdir $fd;
};

my $zip = Archive::Zip->new;
my $total;
my $limit = 5*MB;

my $FileCount = 0;

foreach my $file (@files) {

 my $temp = Archive::Zip->new;

 my $member = $temp->addFile($file);
 next unless $member->compressedSize;

 my $fh = tempfile();
 $temp->writeToFileHandle($fh) == AZ_OK or die $!;
 
 $zip->addMember($member);
 $total += $member->compressedSize;
 #die "$total bytes exceeds archive size limit" if $total > $limit;
 
 my $new_total; 
 if($total > $limit) {
   $FileCount++;
   $total = $total/2;
   print "\n$filecount - $total - $file\n\n";
 }
 #die "$total bytes exceeds archive size limit" if $total > $limit;

}

print "Total archive size: $total bytes\n\n";

$zip->writeToFileNamed('zipped.zip') == AZ_OK or die $!;

[download]
Thanks!

Replies are listed 'Best First'.
Re: Multiple zip files from directories!
by kennethk (Abbot) on Sep 15, 2011 at 15:02 UTC
    The first thing that occurs to me is to add files until you are about to pass 5 MB, and then start a new archive. Perhaps something like (untested): 
    #!/usr/bin/perl -w

use strict;
use Archive::Zip qw/AZ_OK/;
use File::Temp qw/tempfile/;

use constant MB => 1024 * 1024;

my $dir = qw( zip_this/);
my @files = do {
 opendir my $fd, "$dir" or die $! or die $!;
 grep -f, map  "$dir$_", readdir $fd;
};

my $zip = Archive::Zip->new;
my $total = 0;
my $limit = 5*MB;

my $FileCount = 0;
my $archiveCount = 0;

foreach my $file (@files) {

    my $temp = Archive::Zip->new;
   
    my $member = $temp->addFile($file);
    next unless $member->compressedSize;
   
    my $fh = tempfile();
    $temp->writeToFileHandle($fh) == AZ_OK or die $!;
    
    if ($total + $member->compressedSize > $limit) {
        $zip->writeToFileNamed("zipped_$archiveCount.zip") == AZ_OK or
+ die $!;
        $archiveCount++;
        print "Total archive size: $total bytes\n\n";
        $total = 0;
        $zip = Archive::Zip->new;
    }
    
    $zip->addMember($member);
    $total += $member->compressedSize;
}

print "Total archive size: $total bytes\n";

$zip->writeToFileNamed("zipped_$archiveCount.zip") == AZ_OK or die $!;

    [download]
    Update: Fixed copy/paste scoping bug; see Re^3: Multiple zip files from directories! for information. Also initialized $total as per Anonymous Monk's suggestion below.
[reply]
[d/l]
[select]
      This is a great, maybe initialize the variable $total as $total=0; to avoid this:
      Use of uninitialized value $total in addition (+) at test.pl line 34.
[reply]
[d/l]
[select]
      It is making sense now, have one question if you could on this part of the code:
      ...
if ($total + $member->compressedSize > $limit) {
        $zip->writeToFileNamed("zipped_$archiveCount.zip") == AZ_OK or
+ die $!;
        $archiveCount++;
        print "Total archive size: $total bytes\n\n";
        $total = 0;
        my $zip = Archive::Zip->new;
    }
    
    $zip->addMember($member); # <---- This line
...

      [download]
      the last line here where the files are added to the zip file(s), I tried this: $zip->addMember($member) unless $zip->addMember($member); to avoid duplicated files inside of the zip files. But what is happening is that the zip files are been created, that is good, but the files inside of the zip are been added repeatedly. I just need to prevent this somehow. Any suggestions?
[reply]
[d/l]
[select]
        Like I said, untested. The line my $zip = Archive::Zip->new; should have read $zip = Archive::Zip->new;. Rather than creating a new archive in the variable $zip, I was creating a new archive and storing it in a new $zip variable that went out of scope immediately,; hence I kept adding to the old archive. The corrected, functional version is posted in an updated Re: Multiple zip files from directories!
[reply]
[d/l]
[select]
Re: Multiple zip files from directories!
by Anonymous Monk on Sep 15, 2011 at 18:54 UTC

    If you aren't locked into plain old zips, you could simply use winrar or 7zip to make multi-volume archives and then not have to worry about partitioning the files between them.

[reply]
      I can't, cause this will be part of an application, need to use my own code, I am very close now!
[reply]

        Not sure that makes sense to me. If your application can depend on opensource Perl and opensource Archive::Zip, why can't it depend on opensource zip and just system("zip") with appropriate args from Perl?

        I certainly prefer to write Perl instead of execs where possible for performance/reliabilty reasons, but in this case Archive::Zip definitely is not as functional or elegant for doing exactly what you're trying to do here.

        --Dave

[reply]

Back to Seekers of Perl Wisdom