Take a peek at File::Listing's parse_dir function. You should be able to grab the file size by using that. Untested:

use File::Listing qw(parse_dir);

# your ftp code here.
my @dirlist = $ftp->ls("-l");
for (parse_dir(@dirlist)) {
   ($name, $type, $size, $mtime, $mode) = @$_;
   next if $type ne 'f'; # only want plain files
   print "Size of $name is $size\n";
}
[download]