Ok, thanks, i figured it out (sort of). If i just use 
   grep $_, ...

[download]
I get what i want.

The defined($_) wasnt catching some zero-length tokens, but im not quite sure where those zero-length tokens were coming from.

This is for a general address parser for government supplied tax data, which has all sorts of strange formatting in it. I'm turning these strings into a couple DB tables eventually, so that they can be consistently searched.

This part im asking about is tokenizing, then other stuff will identify different parts of the addresses, send it through address correction software, etc.

In reply to Re^2: split and capture some of the separators by shemp
in thread split and capture some of the separators by shemp

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.