I have to do the same thing to about 100,000 documents. Let's say 10,000 files in 10 directories, accessible via FTP. Get them, transform them, FTP them somewhere else.

I'm doing it like this, pseudocode:

for(@folders){
  @list = list of files from folder on ftp server;
  for (@list){
    get file;
    open, do stuff, close;
    ftp somewhere else
  }
}
[download]

But my problem is, FTP isn't that reliable. The connection has crapped out at the "get file" stage a couple of times.

So now all those "or die" tests the Monks have drilled into me don't seem suitable. It's going to take more than 24 hours to do this even if it all goes smoothly.

But when it goes non-smoothly, what would monks do?

• Pause at that point, try to close and re-establish the Net::FTP session?
• Keep going, but make a log of failed files and re-start the whole process, skipping the ones we've successfully got?
• Work by doing some or all of the FTP in one hit, not file by file?
• Look for a tool which would mount the folders rather than using FTP?
• Some other approach?

TIA.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.