It sounds like you have a bad factoring algorithm. Re (tilly) 1: Factoring Decimal Digits shows a much better one.

As for your "pretty cool encryption system", why should I trust an unverified encryption system written by a novice programmer? (Before you get offended by that characterization, learn how to write a link that will work in a browser other than IE.)