Here is my take on your problem.

use strict;
use warnings;
$_ = 'This,is, an, example. Keep $2.50, 1,500, and 192.168.1.1.';
my @words = tokenize($_);
print join "|",@words;

######################################
sub tokenize   
######################################
{
 my $msg = shift;

 my $ntd = qr/(?<=\D)[,.]/;  
 my $dtn = qr/[,.](?=\D|$)/; 
 my $nv = qr/[^A-Za-z0-9\'\$!-.,]+/; 
 
 my %words;
 my @words = grep { !/^$/ and !$words{lc($_)}++} 
               split /$ntd|$dtn|$nv/,$msg;    
 return @words;
}   ##tokenize
[download]

Brief Explaination: split will assume anything matching a certain Pattern to be a delimiter. We have supplied it three patterns (three types of delimiters).

• The first pattern will cause a split anywhere there is not a decimal or comma that is preceded by anything that is not a digit.
• The second pattern will cause a split anywhere there is not a decimal or a comma that is followed by anything that is "not a digit" or the end of line
• The third pattern will cause a split on a series of any chars that you did not did not deem valid.

split then passes this list to grep which will evaluate every member of the list (in this case it checks to make sure that there is something in each item and that the current list item whose lowercase form has not already been seen) and only returns those list values which pass.

Update #1: Another version based on the same regex as above. (I think it looks cleaner):

###############################
sub token_2   
###############################
{
 my $msg = $string;
 my $ntd = qr/(?<=\D)[,.]|[,.](?=\D|$)|[^\w'\$!,.-]+/;
 our %words;
 @words{(split /$ntd/,$msg)} = ();
 return keys %words;
}   ## token2
[download]

update #2:*sigh* I just noticed my second solution looks much like BrowserUK's solution above.

update #3: One more try using m/()/g this time:

################################
sub take_3
################################
{
 my $msg = $string;
 my %words;
 @words{$string =~
   m/(
      (?:
        (?:
           [\w'\$!-]|
           (?<=\d)[.,](?=\d)
        )
       )+
      )/gx}=();

 return keys %words;
}   ##take_3
[download]

-enlil