I agree with the discomfort expressed by sundialsvc4 about doing it at all, but with that caveat, if you do need/want to do it that way, use the SSH facilities to limit the use of the key to come from only a specific IP address, and only allow it to be used to run a specific command.