You're populating @difference with files in the directory that aren't in the log, as well as files in the log that aren't in the directory. Hence deleted files are showing up...

foreach my $logName (keys %loglist) {
        unless (defined $dirlist{$logName}) { push @difference, "$logN
+ame";}
    }
[download]

May be omitted.