perl -e '@files = (0 .. 300_000); system("echo @files") and die $!'
Argument list too long at -e line 1.
[download]

my @escaped = map { quotemeta } @directories;
system("tar cv @escaped|gzip >tarfile.tar.gz") and die $!;
[download]

I'm not sure that this is a Perl problem; tarring and compressing a large number of files -- especially large files -- may well take a long time, regardless of how clever the Perl code is.. How long do the tar and compress operations take to complete? You may get better performance if you can use one of the tar-related modules (see, for example, Archive::Tar), or by springing for a commercial backup utility.

the program is slow and tar process is taking more than half day to complete

I know that this may not be possible but could the system that writes these files be easily altered to write the files into sub-directories? These could be named for a time period, maybe for a day or even for an hour if the file volumes are huge, e.g /dest/dir/2006-12-11 or /dest/dir/2006-12-11.09. Doing this would make administering these files much easier.

I hope these thoughts are of use.

Cheers,

JohnGG

1. Does your version of tar support the "-z / --gzip" (compress) option? If so, you could create the ".tar.gz" file in a single step.
2. Does your version of tar support the "-T / --files-from" option? If so, it would be better to store the items in your @srcfiles array to a simple list file (one file name per line), and pass the name of the list file to the tar command with "-T".
3. Do you really need to use the "-v / --verbose" option when you run tar? That just causes tar to print the names of all the included files to its stdout, but you already have the list of file names, so you don't need this.

open( L, ">$tar_file.input_list" );
print L "$_\n" for ( @srcfiles );
close L;

system( "tar czf $tar_file.gz -T $tar_file.input_list" );
[download]

system( "tar cf - -T $tar_file.input_list | gzip > $tar_file.gz" );
[download]

Apart from that, if you are dealing with lots of data, it's going to take time. Using tar and gzip from the shell command line should give you similar results, and perl has nothing to do with it -- unless, as mentioned in another reply, your list of @srcfiles contains a lot of duplicated entries.

300_000 files is too long an argument list. If all the 300_000 files live within one folder, consider tarring the folder, not the files:

chdir $folder or die "Can't chdir to '$folder': $!\n";
my $ret = system("tar -cvzf $tar_file ."); # tar and zip on the fly
    and die "Couldn't create tarfile '$tar_file': $!\n"; # 'and', not 
+'or'
[download]

If the files live in different folders (e.g. the list was constructed with File::Find or similar) you could split the list into e.g. 1024 item chunks, create the tar file with the first chunk, then update the tar file with the u flag to tar:

my @ary = splice(@srcfiles,0,1024);
my $ret = system("tar -cvf $tar_file @ary") # tar only, no update on '
+z'
    and die "Couldn't create tarfile '$tar_file': $!\n";

while(@ary = splice(@srcfiles,0,1024)) {
    $ret = system("tar -uvf $tar_file @ary");
        and die "Couldn't update '$tar_file': $!\n";
}
[download]

Note also that on some systems performance sinks drastically with big directories (over 100_000 files) while reading file attributes with the stat(2) or lstat(2) system calls (which tar must do to store them), so you would be better off breaking your big folder into smaller ones.

--shmem

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}