Actually the directory is changed, but you don't notice because you are shelling out. That means your code starts a shell, changes the directory within that shell and then the shell exits. That's not what you want.

Use the builtin function chdir. That changes the cwd of the running script. (see perlfunc)