Try adding '.exe' after the 'java' part. Ie:

my $out = system(
    "/users/sophix/Desktop/prottest3/prottest3/java.exe"
    . " -jar prottest-3.0.jar -i $name.aa -t $name.trees -o $name.out"
+ 
    . "-all -verbose -all-matrices -threads 3 -G -F -all-distributions
+ -ncat 4"
);
[download]