I think that WWW-PkgFind might help you.

To quote the docs: "package_retriever scans a web or ftp site for newly posted files and downloads them to a local filesystem. It then prints out the file names of the files it downloads to stdout, suitable for passing to other tools in a pipe". Also, it'll create subdirectories; it uses regular expressions to find "wanted" or "unwanted" files; it can rename packages after download, etc.