Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

I've written a parser using Parse::RecDescent and it works perfect for files of a Not Huge(*) size. But with one of my files the size of one of tokens is just very large.
I've separated that particular token and ran it through 'wc':
940 964 24576
So it's pretty big. When the parser gets to this particular token it segfaults - presumably because the internal perl variable is overflowed (which sounds stupid to me)?

Anyone had this problem before or know something I could do to fix it?

Jay McCarthy
<jay@kenyamountain.com>

Replies are listed 'Best First'.
Re: Parse::RecDescent size limit?
by merlyn (Sage) on Jan 05, 2002 at 21:30 UTC
    P::RD uses a nibbling approach to the text string, and is therefore usually unsuitable for large input streams. theDamian's rumored "Parse::FastDecent" will use m/\G//gc inchworming instead, and be much faster, and probably won't segfault for you. But it's only partially coded, and not yet available, if I recall.

    -- Randal L. Schwartz, Perl hacker

[reply]
      So, I cannot use P:RD for this? Or I should try to use the partial version of P:FD?

      This monk is unenlightened - why does the moon shine on small beings but not large? The know is Buddha.

      Jay McCarthy <jay@kenyamountain.com>

[reply]
Re: Parse::RecDescent size limit?
by Anonymous Monk on Jan 06, 2002 at 01:16 UTC
    Hmmm .... 24576 bytes is really not what I'd call big. 24K, which would represent 0.009% of 256M RAM. I'd guess that merlyn's supposition does not actually apply, though I wouldn't be surprised if merlyn knows somthing I don't in this regard. I've parsed 100M+ files, breaking them into huge chunks (one scalar representing about 10M, 426 times the size of your token, for example) and never had a problem.

    What is the exact error message? If you aren't getting one, you may want to try a trace with something like 

      strace -o my-script.out my-script.pl

    [download]
    and then read up from the bottom of my-script.out to see if there is anything in particular that strikes you just before you see the segfault signal. Just a suggestion ....

    Hope this helps :-)

    P.S. Question for merlyn if he checks back: What is the nibbling approach? I performed a search here and looked through my library, but couldn't find anything ....

[reply]
[d/l]
      P.S. Question for merlyn if he checks back: What is the nibbling approach? I performed a search here and looked through my library, but couldn't find anything ....
      Nibbling is a method of tokenization by constantly attempting substitutions at the beginning of the string, replacing it with nothing if found. 
      $_ = "Hello there, people!";
@output = ();
while (length) {
  (push @output, "word: $1"), next if s/^(\w+)//;
  (push @output, "spaces"), next if s/^\s+//;
  (push @output, "punc: $1"), next if s/^(.)//;
  die "how did I get here?";
}
print map "$_\n", @output;

      [download]
      This is fine for small strings, but the constant leftward shuffling of the string gets very expensive for huge strings. P::RD uses nibbling. P::FastDescent will use pos() scanning instead: 
      $_ = "Hello there, people!";
@output = ();
while (pos $_ <= length $_) {
  (push @output, "word: $1"), next if /\G(\w+)/gc;
  (push @output, "spaces"), next if /\G\s+/gc;
  (push @output, "punc: $1"), next if /\G(.)/gc;
  die "how did I get here?";
}
print map "$_\n", @output;

      [download]
      Much easier, but wasn't available when theDamian first wrote P::RD.

      -- Randal L. Schwartz, Perl hacker

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom