Ok, first, what happens when something you want filtered is split across a file boundary? IMHO (based on the fact that I'm shooting in the dark because you did not post any relevant code) it would be much better to:

  1. Set $/=\5242880 (that's about 5MB).
  2. Open the original 2GB file.
  3. Do a single read (something like my $data = <BIGFILE>; (this would also likely be part of a while loop due to step 7)).
  4. Process the data.
  5. seek back xKB (where x is typically 1 to 2KB; this prevents you from missing something which straddles the read boundary).
  6. Read in another chunk.
  7. Repeat from step 4 until the entire file is processed.

As far as your comment about the process "taking considerable time", we (the Monastary) cannot advise you on whether you are doing the right thing or not unless we see actual code.

In reply to Re: Split 2GB file and parsing by northwind
in thread Split 2GB file and parsing by siva kumar

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.