Re: Re: Spam filtering regexp - keyword countermeasure countermeasure

Hmm, one reply sais to use individual chars, and you use groups of words. I can see how that would work, in that D,E,B and E,B,T are both 3-token groups that will be found.

So I get the feeling that using Baysian analysis on single whole words (e.g. POPFile) is the worst way to do it!

My idea is to add more "context" then POPFile can gleem by itself, by adding special keywords when the preliminary filter spots things.

Comment on Re: Re: Spam filtering regexp - keyword countermeasure countermeasure

Replies are listed 'Best First'.
Re: Re: Re: Spam filtering regexp - keyword countermeasure countermeasure by tachyon (Chancellor) on May 15, 2003 at 01:33 UTC
Combining tokens gives context. It allows you to differentiate to an extent between 'are you free tonight' and 'debt free' 'free widgets' etc. If you just run on 'free' as a word you lose sensitivity as this is quite common. Using consecutive tokens is the way to go IMHO and is the method employed in voice recognition (Dragon used to use 2 words for context and IBM 3 I believe). The price is size and speed based. To give you and idea our single word token file is ~ 100K, the two word token file is ~ 10MB and the three word token file > 1GB You will note a rough 2 order of magnitude increase in size as you add length to the phrases. cheers tachyon s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print	[reply]
Re: Re: Re: Re: Spam filtering regexp - keyword countermeasure countermeasure by John M. Dlugosz (Monsignor) on May 15, 2003 at 19:45 UTC
I suppose that size and speed is an issue for a lot of folks looking for a client-side filter. My machine has cycles to burn while I'm not home and spam keeps tricking in, so it will be all finished analysing when I get around to checking the in-box. So for someone who auto-checks the POP all day, a more server-like solution is feasable. 1GB of disk space is nothing... but weighting down my machine with a background task while I'm doing something interactivly is a bigger deal.	[reply]