Execute traceroute as a read pipe; read its output one line at a time, and close the pipe when the condition is met. To kill the pipe wrap the whole thing in an eval and use an ALRM signal to terminate the process if it takes too long. E.g. (untested)

  open TRACEROUTE, 'traceroute -xyz --whatever|'
    or die "Pipe failed: $!\n";
  {
    local $SIG{ ALRM } = sub { die "timeout" };
    eval {
      alarm ALARM_INTERVAL;
      while (<TRACEROUTE>) {
        # apply your quit test
      }
      alarm 0;
    };
    if ( $@ && $@ !~ /timeout/ ) {
      alarm 0;
      die $@;
    }
  }
  close TRACEROUTE;
[download]

Update: fixed typo: s/script/pipe/.