I did think about that, but i cant quite see how to do it in this case. Part of the problem for me is that the separators are well-defined, but what between them could be anything (except a separator).

You can use the following "continue to split" mechanism in a loop:

(my($token, $sep), $string) = split /PATTERN/, $string, 2;
[download]

This will load the matched string (the separator) into $sep, the stuff before that into $token, and the rest of the string right after the match, into the string, shortening it, ready for the next iteration — provided you have exactly one pair of capturing parens in the pattern.

It's almost identical in effect (bar the negative impact on the global speed of regexes) as using the special variables $`, $&, $' on a normal match, using the same pattern.

If you could have more capturing parens, you can do:

my($token, @sep) = split /PATTERN/, $string, 2;
$string = pop @sep;
[download]

leaving all the captured separators in @sep.

Part of the problem for me is that the separators are well-defined, but what between them could be anything (except a separator).

This looks (to me) like that sentence written in perl:

@list = /([SEPARATORS])+([^SEPARATORS])*/g;
[download]

You could include \s in the second character class if you wanted to ignore whitespace.

Of course, I've been wrong in the past :)

A few (more) examples of what your trying to parse would help, but this seems to do what I think your trying to do?

my @s = ( 
    '129-129A & B-131 NORTH AV', 
    '129-129A + B-131 NORTH AV', 
    '129-129A / B-131 NORTH AV', 
    '129-129A - B-131 NORTH AV',
);
print join'|', grep $_, split '(?:\s+([&/+-])\s+)|\s+', $_ for @s;

129-129A|&|B-131|NORTH|AV
129-129A|+|B-131|NORTH|AV
129-129A|/|B-131|NORTH|AV
129-129A|-|B-131|NORTH|AV
[download]

Update after your update: Maybe this is nearer/

perl> print join'|', grep $_, split '([&/+-])|\s+', $_ for @s;
129|-|129A|&|B|-|131|NORTH|AV
129|-|129A|+|B|-|131|NORTH|AV
129|-|129A|/|B|-|131|NORTH|AV
129|-|129A|-|B|-|131|NORTH|AV
[download]

grep $_,
will erase some tokens such as 0. Better to use
grep length($_),
or
grep { defined($_) && length($_) }

T'is a good point in the general case.

What about just

grep length, split...
[download]

---

Examine what is said, not who speaks.

"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon

Ok, thanks, i figured it out (sort of). If i just use

   grep $_, ...
[download]

I get what i want.

The defined($_) wasnt catching some zero-length tokens, but im not quite sure where those zero-length tokens were coming from.

This is for a general address parser for government supplied tax data, which has all sorts of strange formatting in it. I'm turning these strings into a couple DB tables eventually, so that they can be consistently searched.

This part im asking about is tokenizing, then other stuff will identify different parts of the addresses, send it through address correction software, etc.

but im not quite sure where those zero-length tokens were coming from.

It would appear that if you use capture brackets in a split regex, that $n in returned regardless of whether the capture brackets are in that part of the regex that actually matched. And when they aren't, $1 gets set to the null string ('') rather than undef as you (and I) might suppose.

I've never seen this documented but that seems to be the empirical answer.

---

Examine what is said, not who speaks.

"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon

grep defined($_), split
doesn't work for me.
grep length($_), split
does.

The reason you're getting the zero-length elements is that you have two delimiters that follow eachother. Inbetween those delimiters there's nothing, and therefore you get a string holding nothing.

Example:

    $_ = 'XABX';
    print "<$_>" for split /A|B/;

    __END__
    <X>
    <>
    <X>
[download]

    'X' . 'A' . 'BX' # 'A' matched.
    'BX'             # 'A' removed, 'X' returned.
    '' . 'B' . 'X'   # 'B' matched.
    'X'              # 'B' removed, '' returned.
[download]

ihb