300_000 files is too long an argument list. If all the 300_000 files live within one folder, consider tarring the folder, not the files:

chdir $folder or die "Can't chdir to '$folder': $!\n"; my $ret = system("tar -cvzf $tar_file ."); # tar and zip on the fly and die "Couldn't create tarfile '$tar_file': $!\n"; # 'and', not +'or'

If the files live in different folders (e.g. the list was constructed with File::Find or similar) you could split the list into e.g. 1024 item chunks, create the tar file with the first chunk, then update the tar file with the u flag to tar:

my @ary = splice(@srcfiles,0,1024); my $ret = system("tar -cvf $tar_file @ary") # tar only, no update on ' +z' and die "Couldn't create tarfile '$tar_file': $!\n"; while(@ary = splice(@srcfiles,0,1024)) { $ret = system("tar -uvf $tar_file @ary"); and die "Couldn't update '$tar_file': $!\n"; }

Note also that on some systems performance sinks drastically with big directories (over 100_000 files) while reading file attributes with the stat(2) or lstat(2) system calls (which tar must do to store them), so you would be better off breaking your big folder into smaller ones.

--shmem

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

In reply to Re: Archiving Files by shmem
in thread Archiving Files by mantra2006

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.