sophix:

It looks like you ignored saberworks' suggestion. Your java.exe is probably not in the directory you specified. First change directories, then execute the code. Something like (untested):

chdir("/usrs/sophix/Desktop/prottest3/prottest3");
my @ctl = glob("*.aa");
my ($ctltrimmed, $ctlfile, $ctlfiletrimmed);
foreach my $ctlfile (@ctl) {
    if ($ctlfile =~ m/(\S+).aa$/) {
        $ctlfiletrimmed = $1;
    }    

    my $out = system("java -jar prottest-3.0.jar -i " 
                     . $ctltrimmed . "-t " . $ctltrimmed 
                     . "\.trees" . "-o " . $ctltrimmed 
                     . "\.out" . "-all -verbose -all-matrices "
                     . "-threads 3 -G -F -all-distributions -ncat 4");
}
[download]

...roboticus

When your only tool is a hammer, all problems look like your thumb.