Having looked at the RecDescent man page just now, I'd say
that the use of
the "or" conjunction causes the first element (before
"|") to be tried first, and the latter to be tried only if the
first does not match. In your case, the LITERAL condition
matches not only the content between tags, but also everything
except the "<" of the tags themselves. When you test for
LITERAL first, TAG would never have an oppurtunity to match.
Being less familiar with RecDescent,
I thought it might be easy to make a suitable regex for
use with split() to get the same result as what
you want, but it seems that this alone
would not be enough -- you'd have munge the data a little,
before or after the split, to get your array.
For example, assuming the whole string with tags and content
is now in $_ (and hoping there are no "spurious" angle
brackets), here's a method that inserts split-able strings
around tags to make split produce intended array:
s/(.)</$1=!=</gs; # use some distinct pattern to mark open brackets
s/>(?!=!=)/>=!=/g; # and close brackets that aren't adjacent to "<"
@tokens = split( /=!=/ );
If the string begins with initial "<", the first substitution
won't match that, so we won't get an empty (fictitious) first
element in @tokens. Likewise, the second substitution makes
sure that no empty tokens are generated where the input had
"...><..." (nothing between adjacent tags) or a string-final
">".