Dear Monks, I am trying to open very very large .gz files, but trying to avoid taking a memory hit. Specifically I am running this code:

#!/usr/bin/perl -w
use strict;
use Archive::Tar;

my @here = Archive::Tar->list_archive("filename.tar.gz");
foreach (@here) {
    print "$_\n";
}
print "Took $time_taken seconds to process\n";
[download]

My problem is that the tar.gz file is is 64MB, but the contents are roughly 4GB.

I guess what I should say what I really want. Each of these tar.gz files have just a single file that I need to extract that can be very large. I don't really care about the name inside the tar.gz file, I just want to extract it. Since the file is so large, I would like to extract the file, minimizing the memory hit, so I can then read the extracted contents in, line by line to do the processing that I need to do.

I really want to avoid trying to stick the whole file in memory. Is there a smarter way to do this? Or at the very least minimize the memory hit that I am going to take?

Here is my "best" attempt at this... (this is on a Windows system for *nix tricks won't save me...)

#!/usr/bin/perl -w
use strict;
use Archive::Tar;

my $filename = "filename.tar.gz";
my $tar = Archive::Tar->new($filename);
my @filenames = $tar->list_files;
$tar->extract_file($filenames[0],'temp.out');
open(DATA, "temp.out");
while (<DATA>) {
...
}
close(DATA);
[download]

Thanks in advance!

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.