perlquestion
BigHoss
I need to parse a string which is larger than 2GB. Unfortunately "split()" fails with a "Split loop" error.
<p>
Looking for an alternate. Here is the minimized test case. due to the nature of the original code, the "read()" of the file cannot be changed.
</p>
thanks in advance for any help...<br>
<br>
EDIT: 6/30/2013: <br>
To clarify, this is trimmed down from the original to illustrate the problem with split. <br>
I can only modify "BigParse". I have no control of the strings passed in.
<br>
The real code processes each split until end of data. The print statement in the loop is just to trim down the code.<br><br>
Perl version: 5.12.4<br><br>
Data File is binary file with embedded newline characters "\n".<br><br>
<code>
$ wc -c data
2753110808 data
$ wc -l data
2753111 data
</code>
<code>
#!/usr/bin/perl
$FILE = "data";
open (INFILE, "$FILE") || die "Not able to open the file: $FILE \n";
binmode INFILE;
my $map;
read(INFILE, $map, 2147483648);
# Using this read instead, everything works.
# read(INFILE, $map, 2147483630);
BigParse($map);
exit;
sub BigParse {
my $map = shift;
print "string length = ", length($map), "\n";
# This fails with "Split loop" error message.
foreach my $l (split("\n", $map)) {
print $l;
}
return;
}
</code>