Dave.

Worse than that, I've seen tools explicitly dump something like (...)*? in a regex as something very close to (...){0,32766}?, because repetition only supported 15 bits, not 32, at least in some cases (but maybe that isn't true of modern versions of Perl). But it also seemed like those tools didn't always do such. So I'm not sure how often that limitation applies.

But it is easy to find the breaking point for this particular regex:

 $ perl -del
  DB<1> x 0+( () = join('','<c>','x'x(1<<30),'</c>') =~ m{<c.*?/c>}g )
0  1
  DB<2> x 0+( () = join('','<c>','x'x(1<<31),'</c>') =~ m{<c.*?/c>}g )
0  0
  DB<3> x 0+( () = join('','<c>','x'x((1<<31)-8),'</c>') =~ m{<c.*?/c>
+}g )
0  1
  DB<4> x 0+( () = join('','<c>','x'x((1<<31)-7),'</c>') =~ m{<c.*?/c>
+}g )
0  0
[download]

So (my version of) Perl can't deal with a capture string of more than 2**31-1 characters. And:

 $ perl -del
  DB<2> x 0+( () = join('',('<c>','x'x((1<<30)-10),'</c>')x2) =~ m{<c.
+*?/c>}g )
0  2
  DB<1> x 0+( () = join('',('<c>','x'x((1<<30)-10),'</c>')x3) =~ m{<c.
+*?/c>}g )
0  0
[download]

Surprisingly, it fails to even find the first match if there is a match beyond the 2**31-1 character position? Even trying to iterate to that point doesn't really help (perhaps .*? backtracks?):

 $ perl -del
  DB<1> $x = join('',('<c>','x'x((1<<30)-10),'</c>')x2); while( $x =~ 
+m{<c.*?/c>}g ) { print pos($x), $/ }
1073741821
2147483642
  DB<1> $x = join('',('<c>','x'x((1<<30)-10),'</c>')x3); while( $x =~ 
+m{<c.*?/c>}g ) { print pos($x), $/ }
  DB<2>
[download]

So one needs to deal with the string in reasonably-sized chunks. Which makes me wonder which XML-parsing modules manage to get that right. Their test suites should include a tag with a 4GB attribute value (with an escaped character at the end). :)

- tye        

There you go! That is the shortcoming.

keywords: PERL 64bit(x54)and 32bit(x86) MAXIMUM STRING SIZE REGEX ENGINE 2GB per dave_the_m and focusonz test

keywords: PERL MAXIMUM STRING SIZE 4GB 32bit(x86) ALSO IS WINDOWS X86 LIMIT

keywords: PERL MAXIMUM STRING SIZE 16GB 64bit(x64) ALSO IS Windows 7 Home Premium X64 LIMIT

See http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx#physical_memory_limits_windows_7

You might find it easier to use XML::Twig which is designed to process huge xml files.

Thanks Maybe I will test it next time around

I settled on using a substring() algorithm to parse up the $bigstring which is fast and now processing a total of upwards of 30GB in 5GB chunks.

Whilst loading strings > 4GB is no problem on a 64-bit Perl (assuming you have the memory), unfortunately, there are still many places in the core where such huge strings are simply not supported.

Two examples:

1. substr doesn't accept offsets > 2GB
2. Regexes don't operate on strings > 2GB.

Its a pain in the lower lumbar region, but probably won't change any time soon.

Whoa back!

I am using a construct of if( substr($bigstring, $begtagidx, 5) eq "<c r=" ) Where $begtagidx is out to 4 billion and have not seen problem.

But the data verification process is not yet terminated so I will have to get back to you cloistered people on this.

thanks for the pearls of scripture !

Where $begtagidx is out to 4 billion and have not seen problem.

Okay. It seems that limitation has been lifted with 5.16 (I still use 5.10.1 as my primary Perl where it is the case):

say $];;
5.016001

$s = 'fred'; $s x= 1024**3;;

print substr( $s, -4 );;
fred
[download]

But the 2GB limit on regex still persists in 5.16:

[19:51:25.70] C:\test>\perl64-16\bin\perl \perl64\bin\p1.pl
[0] Perl> say $];;
5.016001
[0] Perl> $s = 'fred'; $s x= 1024**3;;
[0] Perl> ++$n while $s =~ /fred/g; say $n;;
Use of uninitialized value $n in say at (eval 9) line 1, <STDIN> line 
+3.

[0] Perl> $s = 'fr'; $s x= 1024**3;;
[0] Perl> ++$n while $s =~ /fr/g; say $n;;
Use of uninitialized value $n in say at (eval 11) line 1, <STDIN> line
+ 5.

[0] Perl> $s = 'fr'; $s x= 1020**3;;
[0] Perl> ++$n while $s =~ /fr/g; say $n;;
1061208000
[0] Perl>
[download]

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

> Any ruminations on such an eclectic matter?

Regarding 5 GB strings, I wouldn't be surprised if it couldn't be sufficiently held in memory and Perl's attempts to outsource leads to problems.

I am running perl 5.16.2 64bit on a 16GB I7 machine and have given it a cursory test to the 16GB max and have found no other problems

It could be the regex engine in x64 perl still has a 4GB limit?? substring() works out to 16GB $bigstring

Hello focusonz, and welcome to the Monastery!

Do you need @celltags to be fully populated before proceeding? If not, it might be worth trying:

while (m{(<c.*?/c>)}g)
{
    # Process $1...
}
[download]

Just a thought (untested). Hope it helps,

Tried that but took interminably long execution time even on the 1GB string.

The my @celltags = ($bigstring =~ /(<c.*?\/c>)/g); executed quickly on 1GB strings.