Without seeing your code I don't know for sure but I think your problem is not Perl but I/O. I created this test program to create a file with 554,152 100 character records. I then reopen the file and split it into 10 character comma delimited fields using a regexp which I thought would slow. 
use Time::HiRes qw/ gettimeofday /;
use strict;
my $starttime = gettimeofday;

open OUTFILE, ">file.txt" or die $!;
for (1 .. 554152) {
    print OUTFILE "X"x100, "\n";
}
close OUTFILE;

print "Creating file took ", gettimeofday - $starttime, " seconds\n";

$starttime = gettimeofday;

open INFILE, "<file.txt" or die $!;
open OUTFILE, ">file1.txt" or die $!;
while (<INFILE>) {
    chomp;
    print OUTFILE join (",",  /(.{,10})/g), "\n"
}

print "Splitting file using regexp took ", gettimeofday - $starttime, 
+" seconds\n";

__OUTPUT__

Creating file took 8.515625 seconds
Splitting file using regexp took 1.5 seconds

[download]
This is on Win2k/Activeperl 806 using a fast P4 with a 10k rpm hard drive and 1Gb ram so the read is probably all from the disk cache. Check your code and make sure you aren't opening the output file for every line. I've seen people do that and slow things to a crawl.

--

flounder

In reply to Re: What's the most efficient way to write out many lines of data? by flounder99
in thread What's the most efficient way to write out many lines of data? by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.