I need to parse a string which is larger than 2GB. Unfortunately "split()" fails with a "Split loop" error.

Looking for an alternate. Here is the minimized test case. due to the nature of the original code, the "read()" of the file cannot be changed.

thanks in advance for any help...

EDIT: 6/30/2013:
To clarify, this is trimmed down from the original to illustrate the problem with split.
I can only modify "BigParse". I have no control of the strings passed in.
The real code processes each split until end of data. The print statement in the loop is just to trim down the code.

Perl version: 5.12.4

Data File is binary file with embedded newline characters "\n".

$ wc -c data
 2753110808 data
$ wc -l data
 2753111 data

[download]
#!/usr/bin/perl

$FILE = "data";
open (INFILE, "$FILE") || die "Not able to open the file: $FILE \n";
binmode INFILE;

my $map;
read(INFILE, $map, 2147483648);
# Using this read instead, everything works.
# read(INFILE, $map, 2147483630);
BigParse($map);
exit;

sub BigParse {
   my $map = shift;
   print "string length = ", length($map), "\n";
# This fails with "Split loop" error message.
   foreach my $l (split("\n", $map)) {
     print $l;
   }
   return;
}

[download]

In reply to Parse string greater than 2GB by BigHoss

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.