Hi,

I have a large .gz file of 80GB in size to read. Currently I am using the utility use IO::Uncompress::Gunzip to read this .gz file

Each line of the file contains 27,000 entries and I need only particular columns from this source file in the output file. I am now parsing every line in this file using the code, 

while (defined(my $intensities = $INTENSITY_FILE->getline())) {
    my @intensities=split(/\t/,$intensities);
    @final_array=();
    foreach($index=3;$index<scalar(@intensities);$index+=3) {
    push(@index_intensities,($index+1,$index+2));
    }
    push(@final_array,(@intensities[@index_intensities]));
    print OUTFILE join("\t",@final_array);
}

[download]
This will do the task and I am getting an OUTFILE with only the columns I need from the source file. I am looking for another method which could speed up this parsing process and getting the output into a new file. Any comments will be greatly appreciated. Thanks in advance, Praveen.

In reply to Parsing a large file 80GB .gz file and making an output file with specifice columns in this original file. by pillaipraveen

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.