This is just a guess, but... Are you either pre-pending the directory name to the file name, or changing to the directory? The symptom sounds like you're doing something like:

foreach my $dir (@directories)
{
    foreach my $file (@filenames)
    {
        $age = -M $file;
    ....
[download]

When what you should be doing is:

foreach my $dir (@directories)
{
    foreach my $file (@filenames)
    {
        $age = -M "$dir/$file";  # see the difference?
    ....
[download]

or something like this:

foreach my $dir (@directories)
{
    chdir $dir or die "Couldn't chdir to $dir: $!\n";
    foreach my $file (@filenames)
    {
        $age = -M $file;
        ....
    }
    chdir ".." or die "Help, I stranded myself";
[download]

As I said, just a guess. HTH.