I have a few very large text files (~50MB), millions of rows, 5 columns (numbers separated by spaces). The first row is a 2 column 'header'. Also each row ends in a \r\n and every other row is a \r\n on its own. My task was to do something quick and dirty to cut these large files into smaller files. The resulting smaller files would have 300,000 rows per file. I have been learning Perl to deal with just such tasks (I am still working through the Camel book in my 'spare time'). So I tried the following code:

my $pre = $ARGV[0];
my $linenum = 0;
my $filenum = 0;

open FILEOUT, '>', $pre."-".$filenum;

while (<>)
{
  if ($linenum <= 300000)
  {
    if (/^\r\n$/) # skip the linefeed carriage return lines, 
                  #  do not increment line
                  #  counter or print line to file
    {
    }
    else
    {
      print FILEOUT $_;
      $linenum++;
    }
  }
  elsif ($linenum > 300000)
  {
    if (/^\r\n$/)  # skip the linefeed carriage return lines, 
                   #  do not increment line 
                   #  counter or print line to file
    {
    }
    else
    {
      $linenum = 0;  # reset line counter every 300,000 lines
      $filenum++;    #  increment file counter every 300,000 lines 
                     #  and open new file handle
      open FILEOUT, '>', $pre."-".$filenum;
      print FILEOUT $_;
    }
  }
}
close FILEOUT;
[download]

This worked great, I just called the script with each filename on the command line one at a time; except that the new files had 299,701 or 299,702 rows instead of 300,000. I cannot understand how this would happen with the above code! It's really been sand in my shorts, but I bet it is something simple, something a good monk could pick up on! THANKS!

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.