I think that if a module is not used for a long time, its code will just stay on the swap happily, so you should not worry about that unless you have a lack of swap space.