As long as there are no binary compatability issues, you could build and install it to a local directory structure on your workstation that matches the remote box and recursivly ftp it up (ncftp is great for that kind of thing) to the remote box and just make use of it.

It'd be nice if there was some way of building modules localy and 'deploying' them to another box....that might even make a good CPAN module.