http://qs1969.pair.com?node_id=803797


in reply to Re^5: Assistance with file compare
in thread Assistance with file compare

just compare the file sizes. Then only compare (or md5) files whose sizes match.

Not quite. If you need to be absolutely sure the files are identical, the following are effecient ways of achieving this:

  1. Identify files with the same file size.
  2. Of the files with the same file size, identify the files which are identical.

or

  1. Identify the files with the same hash.
  2. Of the files with the same hash, identify the files which are identical.

or

  1. Identify files with the same file size.
  2. Of the files with the same file size, identify the files with the same hash.
  3. Of the files with the same file size and hash, identify the files which are identical.

If you're dealing with many files, the second method is probably the best.
If you're dealing with just a few files, the first method is probably better.

Replies are listed 'Best First'.
Re^7: Assistance with file compare
by ramlight (Friar) on Oct 28, 2009 at 21:26 UTC
    I would think that getting the file size is faster than computing the hash for the file. So it seems to me that pruning the list of files for which hashes have to be computed by comparing file sizes would be faster, especially for large numbers of files.

    I am curious to know why your second method is better for many files. Could you enlighten me please?

[reply]

      I would think that getting the file size is faster than computing the hash for the file.

      You shouldn't be doing either. It should have been done for free when the file was written.

      If you didn't, you could compare files in a clever order and calculate their hash as they are being compared. This may save you from having to do more compares.

      So it seems to me that pruning the list of files for which hashes have to be computed by comparing file sizes would be faster, especially for large numbers of files.

      As the number of files grows, the number of collisions in file size grows.

[reply]

In Section Seekers of Perl Wisdom