Dear Perl Monks,

I'm retrieving large amount of data from a server and after some preprocessing I'm saving it locally in a text file (~3GB). Each time I'm processing a small chunk and I'm wondering if it's better to print the chunk straight after processing, print only let's say after 10 000 chunks (out of ~70,000) or perhaps store them all and print all at once? Speed is the most important part, thought I'm not sure if creating a 3GB array would be wise.

In regards to the data the chunk is split into hundreds of lines. the first line is an identifier and the rest should all by concatenated. I've used split by newline and then .= for that but perhaps join would be better? Or split with a limit of 2 and then use s/\n//g/ on the second part? Again speed is crucial.

Thanks for all the advice.

In reply to Print in loop or store and print once done by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.