curl has a recursive-downloading syntax, not based on spidering but on patterns you specify: http://www.numericals.com/file[1-100:10].txt is a valid curl URL which will download files 1 to 100, incrementing by 10. See http://curl.haxx.se/docs/manpage.html.