Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I have a Perl script that runs on Unix & Windows and it packages large data orders. It needs to be able to create a Tar archive of potentially hundreds of files that will add up to 1 GB or more in total size and it needs to do this without loading the files into system memory.

Initially, I was using Archive::Tar::Streamed, which worked well for small orders, but I eventually ran into Perl's "Out of memory!" message. The CPAN page says, "Tar archives, non memory resident", but I wish I had seen this node ( http://www.perlmonks.org/?node_id=418781 ), first. Now, I'm in a tight spot and I need a solution fast. Can anyone suggest an option?

Many thanks,

Replies are listed 'Best First'.
Re: Lightweight, Platform-Independant Tar
by dragonchild (Archbishop) on Nov 04, 2008 at 01:12 UTC
    For unix, use the tar command. For Win32, use Google. "tar windows" returns an immediately useful response.

    My criteria for good software:
    1. Does it work?
    2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
Re: Lightweight, Platform-Independant Tar
by graff (Chancellor) on Nov 04, 2008 at 01:26 UTC
    GNU tar (the compiled binary that runs with a constant and fairly small memory footprint) is available for windows and is basically native on all unix/linux, so why not use that? (e.g. via a system call)
      While GNU tar might very well build on every Unix, it is reasonably common for it to either not be installed, or to not be called tar. It's often called gtar instead.
Re: Lightweight, Platform-Independant Tar
by sflitman (Hermit) on Nov 04, 2008 at 03:58 UTC
    I've found Archive::Zip to be better when it comes to memory usage, and you can write it to disk quickly and it will then be shorter. To do it all on disk, I agree with the above Monks that you ought to do this in the shell using system. I find zip to be a more resilient format with better error-checking--it is not as good compression as tar/bzip2, or 7zip, but faster than the latter and one-step compared to the former.

    SSF

Re: Lightweight, Platform-Independant Tar
by Anonymous Monk on Nov 04, 2008 at 05:45 UTC
    Thanks to all for your help. My employer has mandated a Tar solution. So, I don't have a choice there. Also, my employer wants the exact same software to run on both UNIX and Windows with nothing more than a configuration file to manage the differences in location for the log files and etc. I'm going to have a look at the GNU Tar application mentioned above.
Re: Lightweight, Platform-Independant Tar
by Anonymous Monk on Nov 05, 2008 at 21:09 UTC
    I installed GNU Tar for Windows from Sourceforge and it works great from the command line. However, when I try to call it from within a simple Perl script I get an error that appears to be saying GNU Tar cannot create the Tar file. Below is the code and the result. Any ideas?

    C:\Temp>type tartest.pl
    use strict;

    $ENV{MYTAR} = "c:\\Progra~1\\GnuWin32\\bin\\tar.exe";

    my ($wrkPath, $testDir);

    $wrkPath = "c:\\temp\\playtar.tar";
    $testDir="c:\\temp\\test";

    chdir($testDir);

    if (system("$ENV{MYTAR} cf $wrkPath .")) {
    print "***Error Tar-ing\n";
    }
    else {
    print "Tar successful\n";
    }

    exit (0);

    C:\Temp>
    C:\Temp>perl -c tartest.pl
    tartest.pl syntax OK

    C:\Temp>perl -w tartest.pl
    c:\Progra~1\GnuWin32\bin\tar.exe: Cannot open c:\temp\playtar.tar: Function not implemented
    c:\Progra~1\GnuWin32\bin\tar.exe: Error is not recoverable: exiting now
    ***Error Tar-ing
    C:\Temp>