In your system command you're single-quoting $file, but not quoting the output file. Adding single-quotes around the output file should solve the spaces-in-filename problem. Alternately, you could "escape" the spaces by putting a backslash in front of each one.

The file extension can be changed with the "substitution" operator s///. (Think search and replace.)

foreach my $infile (@files) {
    print $infile . "\n";
    # escape spaces
    # $infile =~ s[ ][\\ ]g;
    my $outfile = $infile;
    $outfile =~ s/\.(?:mkv|avi)\z/m4v/;
    system "$handbrake -i '$infile' -o '$outfile' --preset=$preset"; #
+ without escaped spaces
    #system "$handbrake -i $infile' -o $outfile --preset=$preset"; # w
+ith escaped spaces
}
[download]