I think what you are looking for is
Perlbot. It has the modularity you are looking for, and is very well written (in my opinion). I tried Infobot as well as another perl based bot called 'bitchbot' and I still much prefer perlbot. Anyhow, go check the link to see if that is what you need. I beleive they are also looking for developer help in getting botnet support in for perlbot if you are intrested.
djw