Three things that I would change:

I wouldn't read the filenames into an array, I'd read them one at a time.

while ($_ = readdir(INDIR)) {
  ...
}
[download]

I'd use next to skip the loop for '.' and '..'.

while ($_ = readdir(INDIR)) {
  next if -d || $_ eq '.' || $_ eq '..';
  ...
}
[download]

Finally (and most importantly) using backticks for commands where you're not capturing the output is wasteful as Perl collects up all the output and returns it to you - only for you to throw it away. Better to use system in that case. Or to use Perl version of the command where they are available - like using move (from File::Copy instead of mv).

Update: One more thing I've just noticed.

unless (-d $srcdir && -d $destdir ) {die "Error: $!";}
[download]

Not sure why you're including the value of $! in that error message. It won't contain a value at that point.