Re^2: Directory comparisons

What happened was there was a connection failure with the remote server and when the script was restarted, some duplication of data occurred and it has sporadically occurred since. Oddly enough, it never happened in operation before the remote server failure. My script retrieves from the remote directory to the local directory, then the files are picked up by other scripts out of my control and placed into the third (also local) directory (among other places). From there they are distributed to customers/websites (scripts also out of my control), so duplication of data is a big no-no. I am considering trying to use the $lc->get_intersection command (List::Compare) to find files that appear in both directory lists, but I'm not sure the best way to check that files here have not already been transferred. Someone has suggested a rotating stand alone file that serves as a "memory" of which files were successfully transferred during a given time frame, but this sounds like a bit of work.

Comment on Re^2: Directory comparisons

Replies are listed 'Best First'.
Re^3: Directory comparisons by zwon (Abbot) on Dec 20, 2011 at 15:22 UTC
but I'm not sure the best way to check that files here have not already been transferred And by transferred here you mean that file was retrieved to local directory, copied to the third directory, and then removed by some other process, is it? In this case yes, you need to keep log of files which were successfully transferred (or rather just list of files that were copied to the third directory).	[reply]
Re^4: Directory comparisons by joeymac (Acolyte) on Dec 20, 2011 at 15:40 UTC
Yes, sorry for the vagueness. That is what I mean by transferred. So the log file(s) should be something like a "dump" of the array contents (directory listing). Then read back in this log file, maybe splitting contents by filename and comparing to the directory list of things to be retrieved. And if it is in the log file, don't transfer it (again). Sound right?	[reply]