Seems like you would want to investigate file recursion here and here. Also, this Q and A deals with recursive processing a single directory.

This deals with the FTP aspects, and this with retrieving online pages.

I hope they are of some use. The Q&A and tutorial sections may also be useful. (I don't have any experience with that module, so I'm not much help there.)