#!/usr/bin/perl -w
use strict;
use Parse::RecDescent;

my $grammar = join( '', <DATA> );

my $parser = Parse::RecDescent->new( $grammar ) or
    die "Error: Bad grammar\n";

my $text =<< "SUB_STDIN";
A nicely [[spaced]] link.
A poorly[[spaced]]link.
Another poorly..[[spaced]]..link.
A harder problem [spaced] link.
Yet harder still..]]spaced[[..link.
The real problem is [[spaced]] followed by ]] or [[ link.
SUB_STDIN

my $results = $parser->startrule( $text ) or
    die "Error: Bad text\n";

__DATA__

startrule: 
    <skip:''> bit(s) 
bit: 
    eol 
    | word 
    | space 
    | token 
    | punct
eol: 
    /\n[ \t]*/ 
    {print "<newline>\n" }
space: 
    /[ \t]+/ 
    {print "< >" }
word: 
    /[\w\']+/ 
    {print "<word: $item[1]>" }
punct: 
    /[^\w\s\[\]]+/ 
    {print "<punct: $item[1]>" }
    | /(?<!\[)\[(?!\[)/
    {print "<punct: $item[1]>" }
    | /(?<!\])\](?!\])/
    {print "<punct: $item[1]>" }
    | /(?<!\[\[)\]\]/
    {print "<punct: $item[1]>" }
    | /(?<!\]\])\[\[/
    {print "<punct: $item[1]>" }
token: link
link: 
    /\[\[(.+?)\]\]/ 
    {print "<link: $item[1]>" }
[download]

--hsm

Hahaha! We're going to beat this grammar into submission yet. :)

Unfortunately we can't brute force it, think of the labor and testing involved to add new tags. I think the best I can do here is to collect punct as single character chunks (storing them in a temp var), then, when I get to a token, insert that temp var back into the tree. I'd post code, but rather than printing discrete sensible morphemes, I really just need the morphemes (productions in P::RD-speak) concatenated in a string. For that purpose whether it emits punct one character at a time or in chunks won't matter.

Either way, this Parse::RecDescent module is the best thing since HTML::TokeParser[::Simple], imho.