system returns -1 on failure to start the program or an error of the wait(2) system call. My guess is that the OP thinks this is returned in $?, since the return value from system is not being checked. Of course, even if the check was correct then it still would not have the desired effect.
I think we agree that checking the return value of system() call is important - same thing like for a file open or whatever. A more important point here could be that using a system() call for the change directory is the WRONG thing to do.
Update: Clarification, because that system call does not change the directory of the Perl script, only the directory of the shell script that the Perl script spawned.