Here is my take on your problem.

use strict;
use warnings;
$_ = 'This,is, an, example. Keep $2.50, 1,500, and 192.168.1.1.';
my @words = tokenize($_);
print join "|",@words;

######################################
sub tokenize   
######################################
{
 my $msg = shift;

 my $ntd = qr/(?<=\D)[,.]/;  
 my $dtn = qr/[,.](?=\D|$)/; 
 my $nv = qr/[^A-Za-z0-9\'\$!-.,]+/; 
 
 my %words;
 my @words = grep { !/^$/ and !$words{lc($_)}++} 
               split /$ntd|$dtn|$nv/,$msg;    
 return @words;
}   ##tokenize
[download]

Brief Explaination: split will assume anything matching a certain Pattern to be a delimiter. We have supplied it three patterns (three types of delimiters).

• The first pattern will cause a split anywhere there is not a decimal or comma that is preceded by anything that is not a digit.
• The second pattern will cause a split anywhere there is not a decimal or a comma that is followed by anything that is "not a digit" or the end of line
• The third pattern will cause a split on a series of any chars that you did not did not deem valid.

split then passes this list to grep which will evaluate every member of the list (in this case it checks to make sure that there is something in each item and that the current list item whose lowercase form has not already been seen) and only returns those list values which pass.

Update #1: Another version based on the same regex as above. (I think it looks cleaner):

###############################
sub token_2   
###############################
{
 my $msg = $string;
 my $ntd = qr/(?<=\D)[,.]|[,.](?=\D|$)|[^\w'\$!,.-]+/;
 our %words;
 @words{(split /$ntd/,$msg)} = ();
 return keys %words;
}   ## token2
[download]

update #2:*sigh* I just noticed my second solution looks much like BrowserUK's solution above.

update #3: One more try using m/()/g this time:

################################
sub take_3
################################
{
 my $msg = $string;
 my %words;
 @words{$string =~
   m/(
      (?:
        (?:
           [\w'\$!-]|
           (?<=\d)[.,](?=\d)
        )
       )+
      )/gx}=();

 return keys %words;
}   ##take_3
[download]

-enlil

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.