in reply to Using Word Tokens as Features
There is a good discussion in the Perl Cookbook that might help you: http://www.en8848.com.cn/reilly%20books/books/perl2/cookbook/ch08_04.htm