As zwon's reply pointed out, the way to speed up file comparisons is to first directly sample the files themselves, say a1k buffer of bytes, then run an md5sum sum comparison ONLY if the buffers match.