There are various "issues" with your code — primarily the following two:

• after the fork, you don't differentiate between the child and the parent process. This means you have two processes both doing the same thing (running the solver, etc.) — the child process with a $pid of zero, and the parent with a $pid unequal zero.

• the backticks do fork/wait themselves under the hood, so the pid you're killing is not the solver process, as intended...

There are several ways to approach this. Personally, I'd probably use a piped open instead of the backticks. Something like this:

sub run_with_timeout {
    my ($cmd, $timeout) = @_;

    local $SIG{ALRM} = sub { die "Timeout\n"; };
    my $out;

    if (my $pid = open my $pipe, "-|") {
        eval {
            alarm $timeout;
            @$out = <$pipe>;
        };
        my $ok = $@ ne "Timeout\n";
        alarm 0;
        if ($ok) {
            close $pipe;
        } else {
            kill 9, $pid;
            waitpid $pid, 0;
        }
        return $out;
        
    } else {
        exec $cmd;
        exit;
   }
}

my $out = run_with_timeout("$solver $file", $timeout);
print defined($out) ? "output of solver:\n@$out" : "timed out";
[download]

P.S.: in case your solver command contains shell meta characters (such as 2>&1), it would be run via a shell. This means you'd have to modify the exec call to read:

        exec "exec $cmd";
[download]

The second exec is executed by the shell, so it replaces itself with the solver command. It is required so that you kill the solver, not the shell.