This is a common problem. readdir returns just the file names, not the path, so you have to prepend the directory name:

my $full_file = "$directory/$file";
[download]

To skip the dot and double dot, just add

next if $file =~ /^\.{1,2}$/;
# or
next if $file eq '.' || $file eq '..';
# ...
[download]

at the beginning of the while readdir loop.

Update: See also File::Find or File::Find::Rule for alternative approaches to walk a directory structure.