Take a look at wget, located at http://www.gnu.org/software/wget/wget.html. Excerpt from that page:

The recursive retrieval of HTML pages, as well as FTP sites, is supported: you can use Wget to make mirrors of archives and home pages, or traverse the web like a WWW robot (Wget understands /robots.txt)