Basically you have to get the job's PID and kill the process.

But since system only returns to your script when the executed command exits, you have to fork first, and in the child do an exec instead of a fork.

You might also find these threads interesting: execute secondary script not waiting for a result and continue with script and particularly Backticks and SIGALRM

(Update: added links; second update: make second link point to root node of the thread)