my %dupes = (); # name->count
my $dref = \%dupes;
[download]

You can, but there is no need. And in this case you're not passing the hash around, so I've left it as a hash, rather than a hash-ref.

Where you DO need a ref is where you store the filedata in %fileinfo. You want an array containing all the files with name X, and you store an array as a hash value by using an array ref.

  @file_info = qw(data1 data2 data3);
  %files = (
     X  => \@file_info
  )
[download]

If you want to add something to the array, then you need to use push, which means you need an array, not an array ref, so you need to dereference it:

  $file_info_ref = $files{X};
  push @$file_info_ref,'data5';

  or

  push @{$files{X}}, 'data5'
[download]

So:

my %fileinfo;
my %dupes;

# for each file, recursing through directory tree
foreach {
   # here $_ is each file name

   # if filename seen already 
   if( exists $fileinfo{$_} ) { 
       # then record in %dupes
       $dupes{$_} = 1;    
   }

   @filedata = ( $fpath, $fsize, $fdate ); # but using real data

   # if fileinfo doesn't yet have an entry for this $_, 
   # then assign an empty array ref
   # saved_info is now an array ref which is 
   # also stored in $fileinfo{$_}

    $saved_info = $fileinfo{$_} ||= [];
    push @$saved_info,\@filedata;

   # No need to re-store it in the hash, because $fileinfo{$_} 
   # and $saved_info both point to the same array
}
[download]

you could shorten

$saved_info = $fileinfo{$_} ||= [];
push @$saved_info,\@filedata;
[download]

to

push @{ $fileinfo{$_} }, \@filedata;
[download]

beacause the arrayref will be autovivified if it didn't already exist.

True. I thought it may have complained that 'undef' (the initial value of $fileinfo{$_}) was not an array ref, but after testing it, I see it doesn't.

thanks, especially for the explanations of 'why' - useful in reducing my ignorance! (I should have said I'd used a reference so it could be passed around to other yet-to-be-done subroutines.)

As far as data structure goes - I would recommend something in the form of:

%files = (
   <md5sum1> => [ <path1>, <path2> ... ],
   <md5sum2> => [ <path1>, <path2> ... ],
);
[download]

HTH,

-- Mickey

It would probably be better to build a HoA keyed by file size ((stat ($file))[7]) rather than MD5 sum, with values being arrays of files of a particular size. Any two files of different size cannot be duplicates, obviously. Any hash element that contained just one file could then be discarded, thus avoiding the expense of MD5 sums or file comparisons for a proportion of the files you are testing.

Once you have sets of files the same size you can compare them either by generating MD5 sums, by reading the files (slurping if small or in chunks if large) and doing string comparisons or by using external commands like cmp. (I would recommend against using external commands.) You can save a lot of time by avoiding re-doing comparisons when you have several files of the same size. For example, given fileA to fileE, you would logically start by comparing fileA to the other four in turn, then fileB to fileC, fileD and fileE, and so on. If fileA differs from fileB but is the same as fileE you can see that it is not necessary to compare fileB with fileE because you already know they differ.

I hope these thoughts are of use.

Cheers,

JohnGG

thanks, that will get binary dupes nicely. I wanted to start with duped filenames and leave binaries for a later pass - I find quite a bit of the music my kids leave around has different binary content but same filenames, but the vice-versa case happens too so I'll add md5 summing as an option. I also want to save the date and size as well as the path+name (hence the array). The bit I've been most unclear on is how to go from finding the first file and setting

  key1 => [ <path1> ]
[download]

to adding the data for the second instance:

  key1 => [ <path1>, <path2> ]
[download]

but hopefully I can put the suggestions above into practise now