If you profile your program you will probably find (after removing the infinite loop that moritz pointed out) that it's spending most of its time waiting for data to be read from or written to the hard drive. File I/O takes a certain amount of time (and is relatively slow) and there's not really anything you can do about it in Perl. In some cases, operating system-level tuning may be able to help with it some, but even that rarely does much, since this is primarily a hardware limitation.

To illustrate this slowness:

$ time cat [a random 700M file] > /dev/null 

real 0m46.649s
user 0m0.020s
sys  0m2.278s

[download]
For a 1G file, we could reasonably expect the time required to be on the order of 67 seconds. And that's just to read the contents of the file and throw them away without doing any processing.

For comparison, try copying your existing file to a new file using your operating system's normal file-copy methods. Since you're doing a copy, plus also doing some other work in the process, then you can reasonably assume that there's no way you can possibly do it faster than a plain copy.

If you're not just encountering an I/O issue, then anything which can be done in your code to speed things up would be inside of the while loop, which you haven't shown us.

In reply to Re: reading file by dsheroh
in thread reading file by saranperl

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.