You might want to ask around at the spam tools mailing list. I used to read it religiously when I was responsibe for maintaining spam filters.

I'm guessing someone's probably already done what you describe. If they haven't, I would probably handle it like soundex, but instead of grouping letters that sound like, grouping glyphs that look alike. (note, I specifically didn't say try to get them to the 'right' value, because the (0Oo) and (1lIi) distinctions are context sensitive ... (100K! M3ds @ lO% 0ff!), and the true meaning doesn't really matter, unless you're trying to determine if it's intentionally obfuscated, as opposed to just a suspicious keywords.)

Oh... and UTF is going to make for a very, very large set of glpyhs.

I agree with jhourcle's words:

(...) distinctions are context sensitive (...)

This is totally true - spammers know this fact and do use it to get around spam filters built this way. One approach we're looking at, tries to use a _capped_ number of replacement sets (ie, perform just 1 (one) to l (ell) transation at a time) and evaluate each of them against the regular expressions.

The results we're getting with this are better than with just regular expressions, but not spectacular. There are more knobs to turn (how many replacements to perform and evaluate, what value should every match add to the score and what is the threshold, for instance) in addition to the set of regexes that are used to detect spam-flag phrases.

A similar approach could be implemented using (hairy, IMHO) regexes. Those regexes are likely much harder to maintain and I guess they might be more expensive than the described approach. However, no testing has been done because we do not have a satisfactory solution to benchmark against yet.

Oh... and UTF is going to make for a very, very large set of glpyhs.

Indeed. This is why you must cap the amount of replacements to do when using this method.

Best regards

-lem, but some call me fokat

HTH!

Whitelisting (only allowing e-mail from known good addresses), can reduce your spam significantly, but it doesn't deal with viruses, and has a rather high rate of false positives (rejecting e-mail that you would have wanted to see ... like maybe that friend from highschool that you've lost track of, or your friend telling you he's been fired from his job and had to switch e-mail addresses)

The only advantage to acting on the e-mail addresses is that it (well, the envelope-sender, not necessarily what shows up in the 'from' header) is sent before the DATA command in SMTP, so you can reduce bandwidth used by rejecting early. (although, that only works for envelope-from and envelope-to ... and I'm guessing unless the system allows <> (the null e-mail address), you're not going to be losing messages about delivery failures.

There are a wide variety of methods for attempting to determine if it's UCE, but most of them tend to only get the obvious stuff, or tend to be over greedy, and block legitimate mail. I agree that some regexes suck, but it takes many, many layers to do it well. (if you're going to go the regex rules, you might start by looking at the procmail rules from panix. I'd also recommend looking at spam-l and spam tools.

I personally find that the best UCE indicator (ie, no false positives, except maybe on spam discussion lists) is when something is obfuscated (octal in IP addresses, HTML w/ hyperlinked urls that don't match the link, javascript to hide the content of the message, etc.)

For most of my mail addresses, only allowing mail from senders in a whitelist, would not be an option, for I do want to receive mails from people that I do not already know. I regularly hand out the email address of our skating crew, and of course we get loads of spam, but also most valuable messages. And as most existing spam filtering mechanism provided by the popular web providers (like Spam Assassin) already filtered out newsletters that I had subscribed to, I only use filtering with a very high treshold, so maybe I have to live with loads of spam in my inbox. Sometimes it is even hard to figure out whether a message is spam or not, when looking at the title and sender as a human being, so how should an algorhythm get this right in every case? I think it is impossible as a matter of principle.

_{fraktalisman keeps rolling}

Both you and jhourcle are right. Relying solely on a whitelist is not the way to go. What I was suggesting was to include that as part of a solution.

I know that with Thunderbird, even though it's using a whitelist, I go through every time I d/l e-mail and see if something got filtered that shouldn't have. The program's learning is part of the solution as well, however it may be beyond the scope of what the op intends.

Useless trivia: In the 2004 Las Vegas phone book there are approximately 28 pages of ads for massage, but almost 200 for lawyers.

Too bad there isn't a way to put Lingua::31337 into reverse gear...

As far as the regex you are looking for, you might get some ideas by looking at some of the rulesets at SARE Rules Emporium. There are many rules that deal with the various obfuscations of messages you're trying to keep out of your email.

Thanks,
Mike

So, what is POPFile, anyway? I couldn't find it on CPAN. Any help would be appreciated. Thanks.

http://popfile.sourceforge.net/

Why trouble to identify the exact words? I think it would be simpler to just have the regex identify obfuscated text, without worrying about exactly what the word is. Look for bizarre arrangements, caps in the wrong places, digits in the words, etc. Maybe keep a list of words that are OK (such as Win98), or (say) module names.

I would also suggest using a whitelist to identify messages from people you know, and pass them along without further processing. It would save you a bit of time, and also make subsequent testing more effective. People you know might use some suspicious words. Strangers should probably not be using obscenities or mentioning body parts in their first message... so it would be OK to filter strongly on those messages.