AFAIK, prime number factorization has little to do with
symmetric encryption algorithms. you are correct when it
comes to asymmetric ciphers (e.g. RSA, DSA), which are
indeed vulnerable to attack based on the fact that the
public and private keys are tied to each other by their
relationship to a particular, large prime; but since
Blowfish is a symmetric block cipher, it is not susceptible
to attacks based on primes.