P.S. Question for merlyn if he checks back: What is the nibbling approach? I performed a search here and looked through my library, but couldn't find anything ....

Nibbling is a method of tokenization by constantly attempting substitutions at the beginning of the string, replacing it with nothing if found.

$_ = "Hello there, people!";
@output = ();
while (length) {
  (push @output, "word: $1"), next if s/^(\w+)//;
  (push @output, "spaces"), next if s/^\s+//;
  (push @output, "punc: $1"), next if s/^(.)//;
  die "how did I get here?";
}
print map "$_\n", @output;
[download]

This is fine for small strings, but the constant leftward shuffling of the string gets very expensive for huge strings. P::RD uses nibbling. P::FastDescent will use pos() scanning instead:

$_ = "Hello there, people!";
@output = ();
while (pos $_ <= length $_) {
  (push @output, "word: $1"), next if /\G(\w+)/gc;
  (push @output, "spaces"), next if /\G\s+/gc;
  (push @output, "punc: $1"), next if /\G(.)/gc;
  die "how did I get here?";
}
print map "$_\n", @output;
[download]

Much easier, but wasn't available when theDamian first wrote P::RD.

-- Randal L. Schwartz, Perl hacker