Some quick searching doesn't reveal any direct CPAN support for connecting to remote tty devices. You could cobble together your own solution any number of ways. One Linux utility that should do the trick easily is 'socat'. Check out the brief instructions here. This would allow Device::SerialPort to talk to a local pty device without knowing everything was being redirected to the remote tty device.

If this works for you, wrapping it all up in a Perl module so that you can easily initiate a connection and even start the remote daemon (via ssh say) would be relatively straightforward.