I just ran a series of experiments. In this case, I'm running on an old linux machine, and copying from a local disk to a nas-mounted directory.

Using File::Copy       : 129 seconds
system("cp blah blah") : 171 seconds
[download]

Now I tried a different approach: @files is a list of the 250 files:

my $cmd = "cd $olddirectory; tar -cvf - " . join(' ',@files) . " | gzi
+p -c -1 | (cd $newdirectory; gzip -c -d | tar -xvf -)";
system ($cmd);

Using gzip -9 : 64 seconds
Using gzip -1 : 57 seconds
Without gzips : 52 seconds
[download]

All times are for copying 250 small files amounting to 1084k. I also found that times scaled symmetrically with number of files.

For what its worth, rsync was copying at the rate of approximately 3 files per second, so the perl/tar method is much faster. (I tried perl in the first place because this directory contains 1.5 million files - rsync was taking more than a day just to collect file info...)

In this case, its apparently more necessary to minimize processing power versus network usage (otherwise gzip -9 would be faster).

As always, your mileage may vary...

Copying a reasonable number of reasonably-sized files should be fast enough by any reasonable method. A bit vague perhaps but the differences are probably too small to measure dependably.

If the files are really big or you have many smaller files, the methods can be compared with Benchmark. Benchmark is normally used to perform many, many repetitions of the same operations in order to gather meaningful statistics. Running many file copy operations could seriously thrash your disk so I wouldn't run any more than a few reps. If that's not enough for to gather stats, you've got your answer (ie, doesn't matter).

Update: This assumes that the folders are on the same computer. Network issues may suggest other methods like tar.

• Windows or *nix ??
• do you really a need a copy of the file?
• are you copying to some kind of storage appliance (SAN?NAS?) what kind of bandwidth do you have?

• Windows: create shortcut from source file in the target directory
• *nix: create link

And I'll tell you why. Unless you are going to scrap the original file, you do not want multiple copies of a file floating around your system. When one changes, you must change all the copies. Unless you document what you are doing very carefully, the coder who has to follow you will not be aware that there are copies or originals that need to be maintained.

Experience tells me that actions taken now will be complete surprises in X number of weeks. Save yourself the bother and embarrassment of unsightly system psoriasis by prevserving what you have and linking it to where you need it to be.

Of course, after careful and mature consideration, you can't see a way around physically copying the files, I think that native utilities (cp, copy) are likely to be the fastest. r,

Augie

I don't know all the reasons why, but I do know that Abigail, world-famous Perl hacker, hates File::Copy. He's presented a Lightning Talk about it (there's a video of the talk).

I just want to make sure you are aware about the issues.

Watching the video, in pieces as it downloads enough to play, I see he uses File::Copy to copy an executable file and it doesn't execute.

That sounds like something odd about the implementation of File::Copy on *Nix, or the underlying file copying OS call it defers to. On Windows, the copy file OS function copies everything including security streams. On the other hand, there is no 'cp' on Windows, so calling system cp won't work either.

So I don't like his final quote about people who don't understand unix tend to reinvent it badly. Instead, people who don't know unix write native file copy API's that work sensibly in the first place.

What I want is a function to copy a file (or directory) that works robustly and has implementations for all platforms. He's arguing that File::Copy isn't it.

—John

How could rsync possibly be faster than the native tools?