I assume you're doing the right thing and using File::Find to find the files in question - then the job can be done at the drop of a hat. The postprocess predicate to find() calls a given sub just before finally exiting a directory. That's all we need - we simply attempt to remove the directory and if there's anything in there, that will just fail harmlessly. It's less work for both us and the computer than if we tried to check manually.

#!/usr/bin/perl -w
use strict;
use File::Find;

my $max_age_days = 120;

find({
    wanted => sub { unlink if -f $_ and -M _ > $max_age_days },
    postprocess => sub { rmdir $File::Find::dir },
}, @ARGV);
[download]

Makeshifts last the longest.