if this is a one time operation Id use rsync or wget -t10 -o 'mylog.log' -c --glob=on --mirror ftp://somesite.com/dir/* or some variation. Id also break up the operation into subs ie get_list, get_file, transform_file, send_file - at each stange you can log errors and restart from there pushing file names into @retry_get_file, @retry_transform, @retry_send_file etc...