Re: Duplicates in Directories

Replies are listed 'Best First'.
Re^2: Duplicates in Directories by haukex (Archbishop) on Oct 09, 2017 at 14:22 UTC
If the file-names come to you in alphabetical order, as they most-commonly do, then all occurrences of baz.anything will necessarily be consecutive. This is not something one should rely on, it varies wildly depending on OS and API used to list the files. And even if one sorted the list, it would not help for the OP's second requirement, "some 'dups' might have minor variations in characters". Oops, didn't see Corion's node before posting.	[reply]
Re^2: Duplicates in Directories by Corion (Patriarch) on Oct 09, 2017 at 14:20 UTC
Whatever you mean by "most-commonly" it is certainly not something that a program should rely on. Luckily for you and the OP, Perl comes with a built-in facility to sort a list of filenames. Using this facility makes it easy to ensure that a list is sorted.	[reply]
Re^3: Duplicates in Directories by Laurent_R (Canon) on Oct 09, 2017 at 18:16 UTC
Well, for a given operating system (and possibly file system), either they do come in alphabetical order, or they don't. If they do for the OP's system, then presumably the OP can rely on that feature (although there can be some issues with the upper or lower case of the file names). And, BTW, I've just checked on the three different systems available to me (nix, VMS and Windows), glob returned the names of the files in the directory in alphabetical system for all three of them, so, yes, it is a rather common feature. Then, of course, as you rightly said, if they don't come in alphabetical order, or if there is any doubt, it is just as easy to use the Perl sort facility, and it will only take a few split seconds with 10,000 files. The idea of sorting data to get better performance (avoiding lookups) is sometimes very efficient. I'm doing it quite commonly in a slightly different context, to compare pairs of very large files that would not fit in a hash: sorting both files on the comparison key (using the nix sort utility), and then reading both files in parallel in my Perl program to detect records missing from either file or differences in attributes of records having the same comparison key.	[reply]
Re^4: Duplicates in Directories by huck (Prior) on Oct 09, 2017 at 18:55 UTC
glob returned the names of the files in the directory in alphabetical system for all three of them, I may be a caveman but id use readdir to get the file list and i think that it returns files in the order they are stored in the directory	[reply]
Re^5: Duplicates in Directories by Laurent_R (Canon) on Oct 09, 2017 at 19:39 UTC
Re^6: Duplicates in Directories by soonix (Chancellor) on Oct 09, 2017 at 20:01 UTC
Some notes below your chosen depth have not been shown here
Re^6: Duplicates in Directories by huck (Prior) on Oct 09, 2017 at 20:28 UTC
Some notes below your chosen depth have not been shown here
Re^2: Duplicates in Directories by Anonymous Monk on Oct 09, 2017 at 14:19 UTC
Auto-logged-out again. Oh well. Just one more thing: remember to do this once more when you reach the end of the file, because that is of course what marks the end of the final group of names.	[reply]
Re^3: Duplicates in Directories by Anonymous Monk on Oct 09, 2017 at 14:28 UTC
Oh yeah, and one more thing: this strategy assumes a case-sensitive file system or that the case of the file-names does not actually vary. Or that you have the easy means to obtain a case-insensitive sorted filename list. But the essential algorithm remains the same. The filesystem’s doing all the hard work for you.	[reply]
Re^4: Duplicates in Directories by Corion (Patriarch) on Oct 09, 2017 at 14:29 UTC
Maybe you could show some actual code solving your assumptions in Perl? That way, you don't need to hope that the users or filesystems behave in the way you describe (because neither does).	[reply]